-
Notifications
You must be signed in to change notification settings - Fork 0
IPFS_Huggingface_Scraper_Py
endomorphosis edited this page Dec 20, 2024
·
1 revision
This module is meant to scrape Huggingface, by using GIT to first collect the metadata from the merkle tree, but instead of actually using GIT-LFS to download the files, it uses the Huggingface-CLI or the Huggingface HTTPS server, so that the files can be downloaded in parallel and without all of the tedious work that is done with git-LFS, and then the files are hashed with and added to the local IPFS node using IPFS_kit_py