IPFS Integration?

[shayaknyc]

Hey folks!

Apologies in advance if this is a silly question, but I've been reading up on IPFS, and how much it's matured over the last few years.

I'd love to somehow integrate my calibre library, and it's growing collection of assets, with IPFS. In a sense, uploading files to IPFS is easy, the problem is, once it's on IPFS, unless the entry has good metadata, you don't really know what file you're downloading, as it's assigned a HASH as its identifier. Like, there's no real search function, unless you use https://ipfs-search.com/.

However, with a proper front-end to manage the metadata of the asset (such as Calibre), which somehow integrates the IPFS back-end, to me, seems like an excellent idea/opportunity - especially if one would like to share their libraries more broadly.

Eager to hear people's thoughts on this....I'm not a developer, so maybe I'm making assumptions on this use-case that don't apply, so please don't chop my head off LOL

Thanks in advance!!

[JSWolf]

I just had a look at what IPFS is and I don't think it will work with Calibre.

[shayaknyc]

Why do you think so? ipfs is basically a distributed storage system....the only real challenge is in being able to find what you're looking for again, once it's been added to the IPFS system. Can you tell me more about your thoughts on why you don't think this would be a good integration? I'm not suggesting the calibre core has to support it, but perhaps a plugin?

LOL Also, scroll down to the section titled "IPFS can help here and now": https://ipfs.io/#how

I mean....it's making a good use-case for calibre...lol

Edit:
Just to make the point, I can very easily, one by one, upload the files/assets I have, and then create a new column in Calibre to track the IPFS CID for each asset uploaded, and make that a link. The problem with doing this, is that it's literally a manual process, and with a library of 2647 entries (and growing) with multiple book formats per entry, this is untenable.

[JSWolf]

Originally Posted by shayaknyc

Why do you think so? ipfs is basically a distributed storage system....the only real challenge is in being able to find what you're looking for again, once it's been added to the IPFS system. Can you tell me more about your thoughts on why you don't think this would be a good integration? I'm not suggesting the calibre core has to support it, but perhaps a plugin?

I can't see anyone wanting to create a plugin to support IPFS. I don't think IPFS will work because Calibre works from a standard drive letter. It does not work on a file system that needs an IP address.

[shayaknyc]

Originally Posted by JSWolf

I can't see anyone wanting to create a plugin to support IPFS. I don't think IPFS will work because Calibre works from a standard drive letter. It does not work on a file system that needs an IP address.

All content that's on IPFS starts off on some local storage - either a drive letter in windows, or a path in *nix/bsd-based OSs - nobody's saving directly to the IPFS, it's gotta get uploaded first, and once it's uploaded, it's provided a hash CID, which is it's unique identifier in the IPFS system, which can then be dynamically linked to via IPNS, so if you ever make updates to an asset, you don't have to update the CID, as the IPNS will point to all revisions/updates of a given CID.

I'm not sure you're making the right argument against somehow leveraging IPFS with one's calibre library. Personally, I'd be very interested in democratizing and sharing my curated collection with whomever would want access to it. It's been wildly successful for scihub (current court-cases not withstanding)......

Edit: I found this quite useful in helping to illustrate my point: https://proto.school/course/ipfs

[DNSB]

Originally Posted by shayaknyc

I'm not sure you're making the right argument against somehow leveraging IPFS with one's calibre library.

I suspect that Jon is trying to say that storing your calibre library on IPFS is not going to work.

Originally Posted by shayaknyc

Personally, I'd be very interested in democratizing and sharing my curated collection with whomever would want access to it. It's been wildly successful for scihub (current court-cases not withstanding)......

As long as you can guarantee that all the books in your collection are not under copyright. If any of them are copyrighted, well, hoist the Jolly Roger and damn the cannons time. Using scihub as an example is not going to help since scihub is mostly referred to as a pirate website which seems to regard copyright, author's rights, etc. as a legal fiction.

Then you run into the books that have a bit of text similar to:

No part of this book may be reproduced in any form or by any electronic or mechanical means, including information storage and retrieval systems.

At MobileRead, even seeming to be promoting piracy is a good way to get yourself banned. MobileRead poobahs prefer to avoid having lawyers casting their eyes in their direction.

[BetterRed]

Originally Posted by shayaknyc

. . .

I'm not suggesting the calibre core has to support it, but perhaps a plugin?

. . .

Optional plugins are almost invariably developed by the person who first identifies the need. Whether and how they make them available to others is their choice. See

• Calibre Manual - Creating Plugins
• Calibre Manual - Plugin API
• Calibre Manual - Setting up a Development Environment

BR

[JSWolf]

Originally Posted by shayaknyc

All content that's on IPFS starts off on some local storage - either a drive letter in windows, or a path in *nix/bsd-based OSs - nobody's saving directly to the IPFS, it's gotta get uploaded first, and once it's uploaded, it's provided a hash CID, which is it's unique identifier in the IPFS system, which can then be dynamically linked to via IPNS, so if you ever make updates to an asset, you don't have to update the CID, as the IPNS will point to all revisions/updates of a given CID.

I'm not sure you're making the right argument against somehow leveraging IPFS with one's calibre library. Personally, I'd be very interested in democratizing and sharing my curated collection with whomever would want access to it. It's been wildly successful for scihub (current court-cases not withstanding)......

Edit: I found this quite useful in helping to illustrate my point: https://proto.school/course/ipfs

In English, this will not work. That's not how Calibre works and it's not how it will ever work. A plugin won't help. You have to change the core code. That's not going to happen unless you do it or find someone to do it.

[shayaknyc]

Originally Posted by DNSB

I suspect that Jon is trying to say that storing your calibre library on IPFS is not going to work.

Hmm....I wasn't suggesting IPFS replace local storage, I was thinking more along augmenting access to the calibre library via IPFS.

Originally Posted by DNSB

As long as you can guarantee that all the books in your collection are not under copyright. If any of them are copyrighted, well, hoist the Jolly Roger and damn the cannons time.

I definitely concede that this is definitely an issue, and I didn't think of it initially as a deal-breaker. I know for a fact that many, if not most, of my library definitely would fall under the category of copyright infringement if it were to become freely and publicly accessible to anyone.

Originally Posted by DNSB

At MobileRead, even seeming to be promoting piracy is a good way to get yourself banned. MobileRead poobahs prefers to avoid having lawyers casting their eyes in their direction.

My intention wasn't at all to promote piracy at all. Thanks for the heads up!

Disclaimer: I AM NOT PROMOTING PIRACY OF COPYRIGHTED MATERIALS THROUGH ONE'S CALIBRE LIBRARY.

This was just a proof-of-concept question. For example, if my entire library was comprised of my own IP, and I would want to make it accessible via IPFS with a Calibre "front-end" to manage the assets, then the rest of this thread applies. I am not condoning that one leverage this for materials that would otherwise be considered piracy.

[shayaknyc]

Originally Posted by JSWolf

In English, this will not work. That's not how Calibre works and it's not how it will ever work. A plugin won't help. You have to change the core code. That's not going to happen unless you do it or find someone to do it.

@JSWolf, I'm not thinking of replacing a storage solution with IPFS, i.e., offloading one's entire library into GDrive or Dropbox. I was thinking more along the lines of extending storage functionality, in addition to one's local storage solution.

[DNSB]

Originally Posted by shayaknyc

Hmm....I wasn't suggesting IPFS replace local storage, I was thinking more along augmenting access to the calibre library via IPFS.

You might want to consider a local calibre library and a one way sync to an IPFS file store. Alternatively, a local calibre library and a save to disk template to stuff more information into the filename and saving to an IPFS file store.

[shayaknyc]

Originally Posted by DNSB

You might want to consider a local calibre library and a one way sync to an IPFS file store. Alternatively, a local calibre library and a save to disk template to stuff more information into the filename and saving to an IPFS file store.

I think my original post may have gotten misinterpreted.

I'm not looking to REPLACE how Calibre stores and manages the library files. I'm looking to extend access to the library assets by way of IPFS as an additional area where these assets may be stored - not exclusively. Sort of like the content-server, or calibre-web, except using IPFS and its ability to definitively identify an asset or tree, by way of its hash, as well as through distributed peer storage.

Since IPFS is still largely in its infancy/tween years, it's difficult to navigate unless you're a pretty advanced user who's familiar with how IPFS is structured. I'm MAYBE an advanced intermediate user, and I'm still learning how IPFS works and don't fully understand it yet. What appears to be missing from IPFS is the ability to search the metadata to identify the asset one is interested in accessing. This is where Calibre's strengths really are - it's a vast database of asset metadata, that's highly searchable. By finding a way to leverage Calibre's existing library management across the IPFS storage space, I thought, would/could be useful.

In essence, I'm thinking it would be possible to somehow link the IPNS for each asset that's uploaded to IPFS, and map it to the asset within calibre. This will allow me to not have to keep track of the IPNS of a given asset, while ensuring the integrity of the data, as well as any updates to that given asset (as the CID would change with subsequent data changes).

[davidfor]

@shayaknyc: I'm pretty sure that your intentions were misunderstood. And reading your last post, I still don't know what they are. That might be because when I looked at your original link, I could not work out what the point was. You pointed to something called "IPFS Search" and that doesn't explain what it is for. I can only see it as a way to search other peoples files and download them. Which immediately raises copyright concerns.

Your last post seems to suggest this is just an alternative to using some other cloud storage service. But, you are also emphasising the search. That basically means I have no idea what you are doing or why.

Having said that...

This seems to be about uploading a file somewhere and keeping a link to it. The latter is simple to do. Calibre has the identifiers. These can be created to link to anywhere. They can be maintained through a metadata source plugin or as an identifier rule. The latter can be created in the preferences.

Or is it about uploading the metadata? In either case, Calibre has extensive support for plugins to extend its function. That is the way to add your syncing function, and generation of the identifiers. The plugin could upload the book or just the metadata in whatever format you needed. Anyone with a some programming skills can create one. It needs to be in Python, but learning that isn't hard and is useful for a lot of things.

[shayaknyc]

@davidfor - apologies, my ADD brain sometimes operates on a million cylinders and it all makes sense in MY head, but sometimes gets discombobulated on it's way out through my fingers.

Baseline premise: local calibre library contains no copyrighted materials (personal intellectual property, open-source materials, or expired copyrights)

What inspired my question was Calibre's in-built content server, as well as the calibre-web project. I like the idea of being able to provide people access to my library, and search its vast database of metadata to identify an asset they'd like to download and read. Calibre's GREAT for this - not just for all the default columns, but for the added columns that work specifically for my use-case, such as DOI (Digital Object Identifier - for journal articles), LOC (Library of Congress call numbers), DDC (Dewey Decimal Classification), DDC Genre (Genre mapped to the DDC number), etc. I also include other metrics with each entry such as the Flesh-Kincaid reading quotient, word counts, etc.

Unrelated to Calibre, I've been reading a lot about IPFS, and it was first on my radar back in the early days of scihub/zlibrary. Within that context, I immediately thought about how I could leverage IPFS to syndicate my curated (copyright-free) calibre library for wider distribution. Keep in mind, I'm not expecting to change anything in the process of adding books and similar assets to calibre - books will still be saved to the "Calibre Library" folder on my local computer.

With this in mind, I couldn't figure a clear and easy way for a file to be added to the IPFS (Inter-Planetary File System) and make it easily discoverable and accessible to someone coming through a federated IPFS client/portal. To further complicate matters, each file gets its own unique CID when added to the IPFS. If there are any updates to that file, the hash will change, and subsequently, change the CID of that asset. To combat this, IPFS has introduced IPNS (which I don't fully understand) - but it effectively points to the same asset as it changes over time.

To bring these together, there are a number of independent projects of individuals making online web search portals for assets that are uploaded to the IPFS. The search leaves....a lot to be desired, and I think it's largely limited by the metadata that's available for each CID/IPNS.

Calibre, by contrast, is very effective at searching the metadata of the assets in its database. Of course, if someone is connected directly to my self-hosted content-server or calibre-web, easy-peasy, they just download directly from my local server, however, there may be cases where a remote user may either have internet connectivity issues, or maybe bandwidth between my local server and the remote user isn't sufficient.

By being able to cross-link the IPNS of a given asset in my library, a remote user would potentially be able to download that book/journal article/calibre entry through the IPFS, which could potentially be hosted by several peers, thereby providing a higher-level of integrity of those assets and their ability to get downloaded safely and securely and guaranteed to be exactly the same file (by way of its hash).

Honestly, now that I've put it down to paper in a more organized way, I may have overcomplicated the whole thing, but it's something that's been brewing in my brain and thought it might be interesting to explore. Likely everyone's gut here could be correct, that it's an unworthwhile endeavor, at least given my specific use-case here.

[rashkae]

I apologize from necroing such an old thread.. But I've recently tacked a similar project. I found this really neat program called calibre2opds that will generate a set of neatly organized web pages of my Calibre library.

I can then add entire calibre folder to ipfs..and browse the library in a webrowser with and address like ipfs://<CID of Calibre Library Folder in base32>/_catalog

I can browse books by tag, author, series, or date range added (for recent books.)