-
Notifications
You must be signed in to change notification settings - Fork 0
IPFS_Transformers_Py
endomorphosis edited this page Dec 20, 2024
·
1 revision
This module overloads the huggingface transformers library, for each of the classes that it provides, and creates a new .auto_download() method, which automatically downloads the files from the fastest source using the IPFS model manager, which supports retrieving the model from the IPFS_model_manager_Py module. The model manager then uses local disk cache, the local IPFS node, S3, libp2p, IPFS_cluster, Huggingface Hub, storacha, and filecoin lassie for retrieval in order the "hotness" the files in question and the storage quotas. During retrieval, it runs a test at downloading readme.md, and whichever source is the fastest will be the one which it tries to retrieve the rest of the data.