Ideas for distributed web archive based on bookmarks creating local snapshots

[DejayRezme]

I'd just like to write down some ideas here, I'm certain that this falls well out of the scope of SingleFile - but maybe it's interesting or useful.

My main problem is that bookmarks have always fallen somewhat short of being useful for me. Even though editing and finding has been vastly improved in firefox it's still a bit clumsy. Worse, half my bookmarks are dead. And sometimes I want to download PDFs so I can load them faster or when I don't have internet.

I now use at least 4 methods to bookmark web content:

1. Bookmark
2. Save page or PDF in user directory / archive HDD
3. Save scientific paper / pdf in Zotero
4. Save webpage as singlefile
5. Save images of designs in some folder structure

So ideally I'd like to merge all these methods into one, just bookmark something and know I'm getting a local copy of everything. If I edit or move the bookmark, the addon scans the bookmark and synchronizes / moves the web cache directory to update it. When I click on a bookmark I can have the option to see the locally stored PDF or web page.

The P2P / distributed web archive idea is not really needed for this, but other people would also start using this functionality, you could store snapshots of the web in a distributed hash table to complement or back up the real archive servers. It could also compress pages for upload to servers which would save them bandwidth and cpu load.

There is also some other intersectionality with this:

6. Save links and notes in markdown document about a specific topic
7. Save links and prices and values in spreadsheet
8. Save doc and 3D printable thing files in zip / unzipped
9. Save data in the cloud

About 6 I use marktext and quite like it. But if I add images I again have a bit of a problem with the ephemeral nature of links on the internet, or having to store and sort them locally which is a hassle and a mess.
And 7 I'd kinda would like a micro-spreadsheet app in something like marktext to do sums of prices etc but of course that has nothing to do anymore with web archiving or bookmarks. Something like zettlr might implement some of these ideas but I haven't even tried that yet.

Anyway, thank you so much for the singlefile addon and just wanted to share some crazy ideas :)

[ryanwwest]

I'm trying to solve a somewhat similar problem as you (also just added this idea a few days back: #1353). For articles/papers I've previously read that are 'good' I want to archive them and save them to Zotero as well to avoid link rot (which itself uses SingleFile under the hood for HTML articles). But for random articles I find and don't have time to read ('read-it-later'), I'd rather be able to read these anywhere (not just on the browser of the device I originally found them) including offline, so I'd like to use SingleFile to save them to a Syncthing folder which then propagates the saved HTML around to all my devices so they're 'local'. They'll likely be deleted after reading, possibly added to Zotero if worth keeping.

I mention this because I don't want to use a dedicated read-it-later app like Wallabag for this, since as you mention, there are many methods of saving content locally and having a different tool for each is unwieldly and for me probably means I won't get in the habit of using it. I do think the fact that Zotero uses SingleFile makes me want to use these in as many places as possible. I doubt your numbers 6-9 would work for SingleFile, but maybe the first 5 could (maybe for images you have a pipeline that extracts all images out of the saved SingleFile HTML and saves them to a content folder, though that might not be granular enough for avoiding unwanted images).

I think a P2P DHT is fine for the idea you propose, if you want to make 'offline' copies more available to others, but haven't seen as much work on DHTs, a few years back many people were trying to use IPFS for sharing archived bookmarks. I haven't looked at this much recently though.

For your comments on 6, I wrote a related article, as I often want to make Markdown notes about some article as well but don't want the source content to disappear. So I save the article to Zotero first (using SingleFile) and then link to that Zotero item in Markdown and Zotero remembers the original URL.

[DejayRezme]

Hey, thanks for your comment and sorry for the late reply. And oh wow, it's insane that we talking about the same workflow and post about literally a day apart!

I think maybe a good solution would be an integration into a hybrid between Zotero and Markdown. So whenever you insert a link to webpage or insert an image, it is automatically cached and store. And also a bookmark is added to firefox and maybe bookmarks are synced. I still have to think more about this.

Also very interesting to hear that Zotero uses Singlefile!

[ryanwwest]

Zotero and Markdown are great since one manages your sources of content (what others wrote) and the other manages your own thoughts (what you wrote). Obsidian has some good plugins for Zotero too to look up anything from zotero and insert a direct link, copy in annotations with links, etc. That might be the hybrid you want though it's not perfect. Zotero can also have multiple attachments for one item - I'm not sure if this could be used (or abused) to store many images. Zotero just started the beta for their android app last month too which can sync attachments with WebDAV.

WEBDAV for SingleFile wasn't working for me though, and may fail for you too if you want to use that.