![]() |
#1 |
Enthusiast
![]() Posts: 42
Karma: 10
Join Date: Sep 2012
Device: Kobo Aura
|
![]()
Hey folks!
Apologies in advance if this is a silly question, but I've been reading up on IPFS, and how much it's matured over the last few years. I'd love to somehow integrate my calibre library, and it's growing collection of assets, with IPFS. In a sense, uploading files to IPFS is easy, the problem is, once it's on IPFS, unless the entry has good metadata, you don't really know what file you're downloading, as it's assigned a HASH as its identifier. Like, there's no real search function, unless you use https://ipfs-search.com/. However, with a proper front-end to manage the metadata of the asset (such as Calibre), which somehow integrates the IPFS back-end, to me, seems like an excellent idea/opportunity - especially if one would like to share their libraries more broadly. Eager to hear people's thoughts on this....I'm not a developer, so maybe I'm making assumptions on this use-case that don't apply, so please don't chop my head off LOL Thanks in advance!! |
![]() |
![]() |
![]() |
#2 |
Resident Curmudgeon
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 75,675
Karma: 134319570
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
|
I just had a look at what IPFS is and I don't think it will work with Calibre.
|
![]() |
![]() |
Advert | |
|
![]() |
#3 |
Enthusiast
![]() Posts: 42
Karma: 10
Join Date: Sep 2012
Device: Kobo Aura
|
Why do you think so? ipfs is basically a distributed storage system....the only real challenge is in being able to find what you're looking for again, once it's been added to the IPFS system. Can you tell me more about your thoughts on why you don't think this would be a good integration? I'm not suggesting the calibre core has to support it, but perhaps a plugin?
LOL Also, scroll down to the section titled "IPFS can help here and now": https://ipfs.io/#how I mean....it's making a good use-case for calibre...lol Edit: Just to make the point, I can very easily, one by one, upload the files/assets I have, and then create a new column in Calibre to track the IPFS CID for each asset uploaded, and make that a link. The problem with doing this, is that it's literally a manual process, and with a library of 2647 entries (and growing) with multiple book formats per entry, this is untenable. Last edited by shayaknyc; 06-17-2022 at 11:21 AM. Reason: add clarity to my vision |
![]() |
![]() |
![]() |
#4 | |
Resident Curmudgeon
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 75,675
Karma: 134319570
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
|
Quote:
|
|
![]() |
![]() |
![]() |
#5 | |
Enthusiast
![]() Posts: 42
Karma: 10
Join Date: Sep 2012
Device: Kobo Aura
|
Quote:
I'm not sure you're making the right argument against somehow leveraging IPFS with one's calibre library. Personally, I'd be very interested in democratizing and sharing my curated collection with whomever would want access to it. It's been wildly successful for scihub (current court-cases not withstanding)...... Edit: I found this quite useful in helping to illustrate my point: https://proto.school/course/ipfs Last edited by shayaknyc; 06-17-2022 at 11:30 AM. Reason: Added link |
|
![]() |
![]() |
Advert | |
|
![]() |
#6 | |||
Bibliophagist
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 38,915
Karma: 153006036
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos
|
Quote:
Quote:
Then you run into the books that have a bit of text similar to: Quote:
Last edited by DNSB; 06-20-2022 at 04:36 PM. |
|||
![]() |
![]() |
![]() |
#7 | |
null operator (he/him)
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 20,855
Karma: 27620684
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Quote:
BR |
|
![]() |
![]() |
![]() |
#8 | |
Resident Curmudgeon
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 75,675
Karma: 134319570
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
|
Quote:
|
|
![]() |
![]() |
![]() |
#9 | |||
Enthusiast
![]() Posts: 42
Karma: 10
Join Date: Sep 2012
Device: Kobo Aura
|
Quote:
Quote:
Quote:
Disclaimer: I AM NOT PROMOTING PIRACY OF COPYRIGHTED MATERIALS THROUGH ONE'S CALIBRE LIBRARY. This was just a proof-of-concept question. For example, if my entire library was comprised of my own IP, and I would want to make it accessible via IPFS with a Calibre "front-end" to manage the assets, then the rest of this thread applies. I am not condoning that one leverage this for materials that would otherwise be considered piracy. |
|||
![]() |
![]() |
![]() |
#10 |
Enthusiast
![]() Posts: 42
Karma: 10
Join Date: Sep 2012
Device: Kobo Aura
|
@JSWolf, I'm not thinking of replacing a storage solution with IPFS, i.e., offloading one's entire library into GDrive or Dropbox. I was thinking more along the lines of extending storage functionality, in addition to one's local storage solution.
|
![]() |
![]() |
![]() |
#11 |
Bibliophagist
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 38,915
Karma: 153006036
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos
|
You might want to consider a local calibre library and a one way sync to an IPFS file store. Alternatively, a local calibre library and a save to disk template to stuff more information into the filename and saving to an IPFS file store.
|
![]() |
![]() |
![]() |
#12 | |
Enthusiast
![]() Posts: 42
Karma: 10
Join Date: Sep 2012
Device: Kobo Aura
|
Quote:
I'm not looking to REPLACE how Calibre stores and manages the library files. I'm looking to extend access to the library assets by way of IPFS as an additional area where these assets may be stored - not exclusively. Sort of like the content-server, or calibre-web, except using IPFS and its ability to definitively identify an asset or tree, by way of its hash, as well as through distributed peer storage. Since IPFS is still largely in its infancy/tween years, it's difficult to navigate unless you're a pretty advanced user who's familiar with how IPFS is structured. I'm MAYBE an advanced intermediate user, and I'm still learning how IPFS works and don't fully understand it yet. What appears to be missing from IPFS is the ability to search the metadata to identify the asset one is interested in accessing. This is where Calibre's strengths really are - it's a vast database of asset metadata, that's highly searchable. By finding a way to leverage Calibre's existing library management across the IPFS storage space, I thought, would/could be useful. In essence, I'm thinking it would be possible to somehow link the IPNS for each asset that's uploaded to IPFS, and map it to the asset within calibre. This will allow me to not have to keep track of the IPNS of a given asset, while ensuring the integrity of the data, as well as any updates to that given asset (as the CID would change with subsequent data changes). |
|
![]() |
![]() |
![]() |
#13 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 24,905
Karma: 47303822
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
@shayaknyc: I'm pretty sure that your intentions were misunderstood. And reading your last post, I still don't know what they are. That might be because when I looked at your original link, I could not work out what the point was. You pointed to something called "IPFS Search" and that doesn't explain what it is for. I can only see it as a way to search other peoples files and download them. Which immediately raises copyright concerns.
Your last post seems to suggest this is just an alternative to using some other cloud storage service. But, you are also emphasising the search. That basically means I have no idea what you are doing or why. Having said that... This seems to be about uploading a file somewhere and keeping a link to it. The latter is simple to do. Calibre has the identifiers. These can be created to link to anywhere. They can be maintained through a metadata source plugin or as an identifier rule. The latter can be created in the preferences. Or is it about uploading the metadata? In either case, Calibre has extensive support for plugins to extend its function. That is the way to add your syncing function, and generation of the identifiers. The plugin could upload the book or just the metadata in whatever format you needed. Anyone with a some programming skills can create one. It needs to be in Python, but learning that isn't hard and is useful for a lot of things. |
![]() |
![]() |
![]() |
#14 |
Enthusiast
![]() Posts: 42
Karma: 10
Join Date: Sep 2012
Device: Kobo Aura
|
@davidfor - apologies, my ADD brain sometimes operates on a million cylinders and it all makes sense in MY head, but sometimes gets discombobulated on it's way out through my fingers.
Baseline premise: local calibre library contains no copyrighted materials (personal intellectual property, open-source materials, or expired copyrights) What inspired my question was Calibre's in-built content server, as well as the calibre-web project. I like the idea of being able to provide people access to my library, and search its vast database of metadata to identify an asset they'd like to download and read. Calibre's GREAT for this - not just for all the default columns, but for the added columns that work specifically for my use-case, such as DOI (Digital Object Identifier - for journal articles), LOC (Library of Congress call numbers), DDC (Dewey Decimal Classification), DDC Genre (Genre mapped to the DDC number), etc. I also include other metrics with each entry such as the Flesh-Kincaid reading quotient, word counts, etc. Unrelated to Calibre, I've been reading a lot about IPFS, and it was first on my radar back in the early days of scihub/zlibrary. Within that context, I immediately thought about how I could leverage IPFS to syndicate my curated (copyright-free) calibre library for wider distribution. Keep in mind, I'm not expecting to change anything in the process of adding books and similar assets to calibre - books will still be saved to the "Calibre Library" folder on my local computer. With this in mind, I couldn't figure a clear and easy way for a file to be added to the IPFS (Inter-Planetary File System) and make it easily discoverable and accessible to someone coming through a federated IPFS client/portal. To further complicate matters, each file gets its own unique CID when added to the IPFS. If there are any updates to that file, the hash will change, and subsequently, change the CID of that asset. To combat this, IPFS has introduced IPNS (which I don't fully understand) - but it effectively points to the same asset as it changes over time. To bring these together, there are a number of independent projects of individuals making online web search portals for assets that are uploaded to the IPFS. The search leaves....a lot to be desired, and I think it's largely limited by the metadata that's available for each CID/IPNS. Calibre, by contrast, is very effective at searching the metadata of the assets in its database. Of course, if someone is connected directly to my self-hosted content-server or calibre-web, easy-peasy, they just download directly from my local server, however, there may be cases where a remote user may either have internet connectivity issues, or maybe bandwidth between my local server and the remote user isn't sufficient. By being able to cross-link the IPNS of a given asset in my library, a remote user would potentially be able to download that book/journal article/calibre entry through the IPFS, which could potentially be hosted by several peers, thereby providing a higher-level of integrity of those assets and their ability to get downloaded safely and securely and guaranteed to be exactly the same file (by way of its hash). Honestly, now that I've put it down to paper in a more organized way, I may have overcomplicated the whole thing, but it's something that's been brewing in my brain and thought it might be interesting to explore. Likely everyone's gut here could be correct, that it's an unworthwhile endeavor, at least given my specific use-case here. |
![]() |
![]() |
![]() |
#15 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,211
Karma: 5372421
Join Date: Jun 2011
Location: Ontario, Canada
Device: Kobo Aura HD
|
I apologize from necroing such an old thread.. But I've recently tacked a similar project. I found this really neat program called calibre2opds that will generate a set of neatly organized web pages of my Calibre library.
I can then add entire calibre folder to ipfs..and browse the library in a webrowser with and address like ipfs://<CID of Calibre Library Folder in base32>/_catalog I can browse books by tag, author, series, or date range added (for recent books.) |
![]() |
![]() |
![]() |
Tags |
ipfs |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Goodreads Integration | mmobes | Library Management | 0 | 02-05-2022 02:24 PM |
Calibre integration | NightStar76 | PocketBook | 4 | 11-24-2020 04:50 AM |
integration with Hook | adrianf | Calibre | 16 | 06-25-2020 11:30 PM |
Libra integration with pocket ? | stumped | Kobo Reader | 5 | 12-22-2019 06:45 PM |
Dropbox integration | aceflor | Kobo Reader | 8 | 10-07-2019 04:03 PM |