Skip to content

IPFS_Huggingface_Scraper_Py

endomorphosis edited this page Dec 20, 2024 · 1 revision

This module is meant to scrape Huggingface, by using GIT to first collect the metadata from the merkle tree, but instead of actually using GIT-LFS to download the files, it uses the Huggingface-CLI or the Huggingface HTTPS server, so that the files can be downloaded in parallel and without all of the tedious work that is done with git-LFS, and then the files are hashed with and added to the local IPFS node using IPFS_kit_py

Clone this wiki locally