Setting Up a Calibre Library on an SD Card

[Epsilon Rose]

I've decided to, finally, get around to downloading and organizing the collection of e-book bundles I've purchased from Humble and I'd like to use Calibre to help with that. I also think I know what I'd like the final result to look like, but I'm not sure how to get there and I'm hoping someone on here can help me. Here's what I'm trying to do:

• I want my entire library to live on a microSD card. (I explicitly do not want two copies of my library, with one on the PC and one on the card.)
  • The card will move between my PC and Boox TUC e-reader, based on what I'm doing, but it will likely spend most of its time in the e-reader.
• I do not want to rely on a network connection to my PC while I'm reading on my e-reader.
• I'd like to use the portable version of Calibre, so it can live on the same card, but I'm extremely flexible on this point, if there's a decent reason not to.
• I'm planning to use Moon+ Reader and Boox built in reader on the TUC, but I'm open to other apps, even paid ones, so long as they don't require a subscription.

Now for the tricky part. Here's how I'd like to interface with my library:

Reference Books
-> Subject
-->Sub-catagory

Graphics Novels
->Manga
-->Series
->Western Comics
-->Publisher
--->Series/Event/Character
---->Sub-Series
Novels
->LN
-->Series
->Western Novels
-->Author
--->Series

Now, I know Calibre relies on tags and filtering, rather than a folder structure, and it's pretty easy to see why. Unfortunately, that only works if you have access to the tag and filter functionality, and I don't think Moon+ Reader does. Without it, reverting to author->title is pretty cumbersome. To my knowledge, the main ways to deal with this are to either A) Export your library, which allows for an arbitrary folder structure, or B) Remotely connect to Calibre to handle sorting and retrieving the books.

Unless I'm misunderstanding something, neither of these solutions really work for me. A requires maintaining two separate copies of the library, and duplicating all of the books, while B) seems to rely on a remote connection to my computer and, possibly, also duplicating the books. Is there another way around this dilemma or something I'm missing?

To be clear, I don't have a problem with Calibre maintaining it's own folder structure and using the same kind of tag/filtering on my TUC would be an improvement, but I don't know if that's possible. Is there a way to get Moon+ Reader to present my books in a more accessibly structure, without relying on the folders (possibly based on a file Calibre hands it or meta data) or is there a better app I should be using?

Also, any other tips on setting up this kind of mobile library or dealing with Humble would be appreciated.

[Quoth]

Originally Posted by Epsilon Rose

The card will move between my PC and Boox TUC e-reader, based on what I'm doing, but it will likely spend most of its time in the e-reader.

No, that's not how Calibre works. The ebook files only exist as ebook files because it's not viable to import the actual ebook content into an SQL database, esp. SQLight, so the actual SQL database file has the ebook file information. You need to treat the ebooks imported to Calibre as if they are invisible, except for backups.

There is a Calibre "Save to Disk" and that should be used with the SD card for the Ereader if not connecting it via USB. I connect my phones, tablets, ereaders etc via USB direct to Calibre and manage them. I backup the Calibre "system" including the private ebook files with rsync when Calibre is not running.

I've also tested the OPDS via Content Server and WiFi on one tablet running Pocketbook, but USB is simpler.

With Save to Disk, you can use templates to have any filestructure you want. But any ereader App (Moonreader, KOReader) that only uses a file system interface is like 1960s computing and gets very awkward with more than a few hundred titles. I found KOReader nearly unusable with 6,500. Also file organisation can't do collections (files would have to be duplicated) or series or multiple author titles easily.

You can't change how Calibre stores the imported ebooks, because those are ONLY for Calibre, they are part of the database.

I have about 7,000 ebooks and I'm happy with how Calibre works. Some people may have 60,000 or many more (maybe they can read half a dozen books a day or will live for 100s of years?).

The SD card has to have an export from the library, it's not the library, and you can use template to have the structure you want for Moonreader.

[Quoth]

Yes, Moonreader isn't the best. Lithium and Pocketbook apps are far better (even free Pocketbook, uncheck 2nd box at install time). If the ebook needs tweaks, fix it in Calibre Editor. You can check an added ebook quickly with Calibre viewer and fix something faster than fiddling with over-ride settings in an ereader app.

Calibre is Library, device manager, viewer, editor, search system (inc full text search), ebook creator from docx and conversion program. The ebook files, once added are "private", hence being able to treat a directory/folder/disk as a device for "Send"/export/save.

[Epsilon Rose]

Originally Posted by Quoth

No, that's not how Calibre works. The ebook files only exist as ebook files because it's not viable to import the actual ebook content into an SQL database, esp. SQLight, so the actual SQL database file has the ebook file information. You need to treat the ebooks imported to Calibre as if they are invisible, except for backups.


There is a Calibre "Save to Disk" and that should be used with the SD card for the Ereader if not connecting it via USB. I connect my phones, tablets, ereaders etc via USB direct to Calibre and manage them. I backup the Calibre "system" including the private ebook files with rsync when Calibre is not running.

Ok ... But those "invisible" files still take up disk space. They'll take up the same amount of disk space as the actual "read" copies of the files. That may not sound like much, but when you're dealing with graphic novels and manga that you want to be decently high quality, it starts to add up pretty quickly. I don't want to dedicate that much space to my library on my computer's built in drive, especially when I won't be doing most of my reading on said computer.

I'm also aware of the "Save to Disk" feature; it's what I was referencing when I mentioned the export feature. (Sorry for the mistake.) The problem with handling it this way is it, functionally, doubles the size of my library, which I'd rather not do for what I hope are obvious reasons.

With Save to Disk, you can use templates to have any filestructure you want. But any ereader App (Moonreader, KOReader) that only uses a file system interface is like 1960s computing and gets very awkward with more than a few hundred titles. I found KOReader nearly unusable with 6,500. Also file organisation can't do collections (files would have to be duplicated) or series or multiple author titles easily.

I feel like you have an unrealistic view of how computing worked in the 60s ... or how people used computers for most of the time since then.

The SD card has to have an export from the library, it's not the library, and you can use template to have the structure you want for Moonreader.

Unless the "library" can not have the files for the books, and just stores data about them, the SD card needs to have the library.

Originally Posted by Quoth

Yes, Moonreader isn't the best. Lithium and Pocketbook apps are far better (even free Pocketbook, uncheck 2nd box at install time). If the ebook needs tweaks, fix it in Calibre Editor. You can check an added ebook quickly with Calibre viewer and fix something faster than fiddling with over-ride settings in an ereader app.

Unless I'm looking at the wrong versions of Lithium and Pocketbook, both of those seem to have some very serious shortcomings.

Lithium seems to only read EPUBs or, at least, they don't bother to list compatibility with anything else. And, while Pocketbook does have compatibility with all of the file formats I need (epub, pdf, cbr, cbz, probably a few other random ones), it doesn't seem to have any meaningful tools for actually viewing your library. As far as I can tell, if you aren't viewing it by file structure, you're limited to filtering by file format, looking at manually created collections, or free-form text search. It can, apparently, sort by series, but you can't browse by series or see a list of series. I'm certainly not seeing any sort of tag filtering.

[DNSB]

Originally Posted by Epsilon Rose

I feel like you have an unrealistic view of how computing worked in the 60s ... or how people used computers for most of the time since then.

I used computers in the 60's. In the 70's I had my own home computers (4300 solder connections and you too could have your own personal computer). For me with a calibre library of ~16,000 ebooks, metadata search (title, series, author, etc.) is a necessity. Even when I had <500 books back in the 2010's, try to tailor a directory structure that made it easy to find books was a total pain. And yes, storing files in directory structures with no metadata is so 80s.

Originally Posted by Epsilon Rose

Unless the "library" can not have the files for the books, and just stores data about them, the SD card needs to have the library.

Given that in my current computer collection, the smallest hard drive is my Raspberry Pi with a 256GB NVMe drive and my 16,000 books take 41GB of space, I see no issues with keeping my calibre library local.

As for your worries about two libraries, the calibre library would be attached to your main computer. The SD card would be inserted into the computer and any new ebooks could be copied to the SD card. No need to have two copies on your computer.

You could also look at KOReader since it can use the calibre metadata file to search. You could set up your SD card as an device using Preferences => Tweaks => search for folder. Add the path to your SD card folder that will be used (i.e. F:/mylibrary where the SD card is mounted as drive F: and you want to use the mylibrary folder for your books. Update the ebooks on the card and move it back to your Boox.

[Epsilon Rose]

Originally Posted by DNSB

I used computers in the 60's. In the 70's I had my own home computers (4300 solder connections and you too could have your own personal computer). For me with a calibre library of ~16,000 ebooks, metadata search (title, series, author, etc.) is a necessity. Even when I had <500 books back in the 2010's, try to tailor a directory structure that made it easy to find books was a total pain. And yes, storing files in directory structures with no metadata is so 80s.

I never said I didn't want to use metadata. In fact, I was pretty explicit that it was an obviously useful way to handle things, provided you can actually make use of it.

Given that in my current computer collection, the smallest hard drive is my Raspberry Pi with a 256GB NVMe drive and my 16,000 books take 41GB of space, I see no issues with keeping my calibre library local.

Ok. I'm going to guess those are mostly text based books. If you want to read high quality comics, then storage space gets eaten up much more quickly than that. To use real world numbers, downloading a single instance of every comic in the Best of BOOM comic bundle takes up about 13GB. I have more than 10 graphic novel bundles, even before I get in to the actual books or TTRPGs. I don't want to dedicate over 100GB of storage to them on a device where I don't intend to read them very often, especially when I already have other things competing for that space.

As for your worries about two libraries, the calibre library would be attached to your main computer. The SD card would be inserted into the computer and any new ebooks could be copied to the SD card. No need to have two copies on your computer.

I didn't say two libraries on my computer. I said two libraries. The one on my computer is the one I don't want.

You could also look at KOReader since it can use the calibre metadata file to search. You could set up your SD card as an device using Preferences => Tweaks => search for folder. Add the path to your SD card folder that will be used (i.e. F:/mylibrary where the SD card is mounted as drive F: and you want to use the mylibrary folder for your books. Update the ebooks on the card and move it back to your Boox.

I've seen KOReader recommended before, but unless I'm missing something, it seems like the only way it allows you to browse your library is via the folder structure?

[Quoth]

Originally Posted by Epsilon Rose

I've seen KOReader recommended before, but unless I'm missing something, it seems like the only way it allows you to browse your library is via the folder structure?

Yes, that's the only browsing. But it can search the Calibre Metadata file. A bit clunky, so I read novels on a Kobo using the Kobo metadata library to browse as it's perfectly supported by Calibre. Everything else is dramatically poorer and worst is file browsing.

Kobo:
Filter by read, reading or unread.
Sort by title, recent, collection, series, author etc.
Subtitle display

Search is faster than KOReader and lists hits of title, series, author, subtitle.
About 6,700 novels on my Kobo.

I can't imagine what you think the shortcomings of pocketbook app are.

My tablet has a 256 G SD card and that has a copy of every document since about 1994, including 1000s of PDF scans of books, magazines and service information (mostly late 1800s to 1980s).

My old 2016 Laptop is 1T byte. My current laptop is 256 G SSD and 2T HDD. Workstation is 512G SSD and 4T 3.5" HDD.
Storage is cheap.
Several backup drives 1T to 6T.
Old server for backups is 1T + 2T drives.

I only used mainframes 1969 to 1973 and then personal micros from 1979.

[Quoth]

Originally Posted by Epsilon Rose

I didn't say two libraries on my computer. I said two libraries. The one on my computer is the one I don't want.

Calibre runs on Mac, Linux and Windows. Smallest HW an ARM based Raspberry Pi with Liinux.
It uses a local disk, only. At a pinch you can use a USB drive or SD card that's formatted with the native filesystem of the OS Calibre is using.

You can only use an exported copy of the files (which is NOT a Calibre Library) without the Calibre program.

You can't run Calibre without the native local Calibre files. Even a network share will give grief. Calibre does have a content server to export the content of the local files on a network.

You can only use a Tablet or an SD card for a tablet (or phone or ereader) as a destination for Calibre to send copies to.

My Raspberry Pi4B does have Calibre and only has an SD card for storage, but that card can't be sensibly used in anything else, except maybe a compatible replacement RPi. It does have a portable QHD screen and power adaptors to run without mains (2 x 2A USB power banks and/or a 12V 8AH Gel pack / SLA with 2x 5V USB converters and a nice Cherry 10KLish keyboard.

[Epsilon Rose]

Originally Posted by Quoth

Yes, that's the only browsing. But it can search the Calibre Metadata file. A bit clunky, so I read novels on a Kobo using the Kobo metadata library to browse as it's perfectly supported by Calibre. Everything else is dramatically poorer and worst is file browsing.

Kobo:
Filter by read, reading or unread.
Sort by title, recent, collection, series, author etc.
Subtitle display

Search is faster than KOReader and lists hits of title, series, author, subtitle.
About 6,700 novels on my Kobo.

I'd like to point out that both Moon+ and Librera can filter by series, authors, tags, and several other things. Librera can also search by those things, though I'm less sure about whether or not Moon+ can search by all of those things and I don't know how either of them interact with Calibre's meta data specifically.

I can't imagine what you think the shortcomings of pocketbook app are.

I can't seem to find any way to filter your library by tags or what not. Search might work, but that's not always as convenient as filtering, especially when it seems to lack any form of auto complete. This might be due to the files I'm testing it with not coming from Calibre, but other reader apps seem to be able to pick up on their metadata just fine.

Actually, this touches on a major shortcoming that I've found with all of the apps I tried. Some of them let you filter your view based on various criteria (e.g. tag, author, series), but none of them seem to allow you to filter based on on multiple criteria, simultaneously, without resorting to a manually typed search. Similarly, they don't seem to have any ways to group how results are displayed after you do that filtering or sorting. For example, I might want to filter by Medium: Graphic Novels and Author: Neil Gaiman, and then view the results by series, rather than individual book. That way I could see all of the graphic novels series I have by Neil Gaiman and then I could select a specific series to see the individual issues I have. This seems like exactly the type of flexibility that tagging is supposed to allow vis-a-vis a folder structure based system, but it doesn't seem to actually be available anywhere. Am I missing something?

My tablet has a 256 G SD card and that has a copy of every document since about 1994, including 1000s of PDF scans of books, magazines and service information (mostly late 1800s to 1980s).

My old 2016 Laptop is 1T byte. My current laptop is 256 G SSD and 2T HDD. Workstation is 512G SSD and 4T 3.5" HDD.
Storage is cheap.
Several backup drives 1T to 6T.
Old server for backups is 1T + 2T drives.

I only used mainframes 1969 to 1973 and then personal micros from 1979.

I'm not entirely sure what this is supposed to prove? I'm glad your set up works well for you and lets you store a large number of documents, but I provided the real world numbers for my use case. They may be different than what you're working with, but that doesn't make them less valid. Similarly, just because I prioritize how I use my internal storage space differently does not mean my conclusions are invalid and I don't think "100GBs is too much space to dedicate to my library on my main PC" is a particularly strange conclusion. Storage may be cheap, but it's not free and laptops only have so many drive bays and usb ports available.

Originally Posted by Quoth

Calibre runs on Mac, Linux and Windows. Smallest HW an ARM based Raspberry Pi with Liinux.
It uses a local disk, only. At a pinch you can use a USB drive or SD card that's formatted with the native filesystem of the OS Calibre is using.

I do have an old pi just sitting in a box and I've been trying to figure out a project for it for a while, so I may try doing something with that, but that will add enough additional steps and complications to the project that I wouldn't want to start with it or even attempt it right now. This is the sort of thing I might do because I enjoy playing with electronics, not because it's a practical solution to my problem.

I find it strikingly strange that the idea of storing a library on an SD card, moving it between devices, and using an app to interface with it on Android is seen as such an impossible use case. Especially because, at the end of the day, all it really requires is an app that can interact with a library in the same way Calibre does or just reference the files Calibre already generates to do the same thing. The fact that Calibre already comes in a portable version means that someone already recognized that this is something users might want to do. It's just that the current implementation is limited to PCs, for some reason.

[DNSB]

Originally Posted by Epsilon Rose

I find it strikingly strange that the idea of storing a library on an SD card, moving it between devices, and using an app to interface with it on Android is seen as such an impossible use case. Especially because, at the end of the day, all it really requires is an app that can interact with a library in the same way Calibre does or just reference the files Calibre already generates to do the same thing. The fact that Calibre already comes in a portable version means that someone already recognized that this is something users might want to do. It's just that the current implementation is limited to PCs, for some reason.

That virtually all the ePub renderers I've looked at show many signs that the developers' suffered from NIH (not invented here) and insisted on doing everything their way. IMNSHO, Moon+ is a terrible product since it has little if any respect for publisher CSS even if you manage to find the toggle to enable publisher CSS never mind worrying about interoperability with other renders. But then I have the same opinion about many of the Android and iOS ePub renderers.

That no one has managed to create an Android version of calibre might be an indication that it's not as trivial a task as you seem to believe. Some of the >2200 forks of calibre were attempts to port to other operating systems including Android. None of the ones I looked at got to a gamma version much less code stable enough to release. The old saw that "Every programming task is easy to a non-programmer" comes to mind.

The portable version of calibre differs from the non-portable version more in the number of Windows variables set by it's startup bootstrap before launching the calibre executable than anything else. For Linux and MacOS, it is much easier to install and run calibre from a random location.

[Epsilon Rose]

Originally Posted by DNSB

The portable version of calibre differs from the non-portable version more in the number of Windows variables set by it's startup bootstrap before launching the calibre executable than anything else. For Linux and MacOS, it is much easier to install and run calibre from a random location.

The portable version is also explicitly designed to be self contained, including using a library in the portable folder, rather than anywhere you might want one.

[Quoth]

Originally Posted by Epsilon Rose

I find it strikingly strange that the idea of storing a library on an SD card, moving it between devices, and using an app to interface with it on Android is seen as such an impossible use case.

The Calibre Library = Calibre program (to access) + Calibre database SQLight file + "private" ebook files. It needs to use the native Filesystem for the internal copy of the ebooks.

An SD card for a Tablet, phone or ereader may not support that Filesystem. Also you need to unmount/mount the SD card to move it between devices and Calibre is broken when you eject / unmount the SD card, if it did have ext4, HPFS (or whatever Apple Mac uses today) or NTFS.

If you want the ebooks in Calibre (only Mac, Linux and Windows) and on a tablet/phone/ereader the solution is NOT using an SD Card for Calibre. The solution is managing the device via USB, or managing and SD card for the device byy Save/Export.

I mentioned irrelevant stuff to illustrate I am not talking rubbish. I've designed and maintained databases and document management systems for decades and doing what you want with an SD card that has Calibre's "private" database of ebooks isn't absolutely impossible if the ereader/tablet/phone can read the computer's native FS format (NTFS for Windows or ext4 for Linux), but it's very foolish. If SQL could sensibly internally store the ebooks there would be no separate "private to Calibre" ebook files, but the space would STILL be used to have internal copies.
A true library / document management system needs a private copy of the media files, and you send/copy/save/export copies for other systems like phone/tablet/ereader.

I do have the same Calibre library on more than one computer, but one is the "master" and I sync copies to the others when calibre is not running on either and changes are only made on the master version of Calibre. The others are only used to view.·

[Quoth]

If database implementations were perfect you would never see the files for ebooks added to Calibre, or they'd be in the database.

Even if there was an Android version of Calibre you couldn't safely move an SD card between that and a Windows PC version of Android.

The "Portable" version of Calibre is really just for a Windows PC that doesn't have Calibre actually installed. It's Windows version of Calibre program, the SQL database and the "private" ebooks. It's no use to Mac, Linux (they don't need it anyway), iOS or Android.

A true Library is the database, the private media files and the program.

[DNSB]

Originally Posted by Epsilon Rose

The portable version is also explicitly designed to be self contained, including using a library in the portable folder, rather than anywhere you might want one.

I use calibre portable when I'm checking differences between versions and I can browse to my existing library and use it. Usual cautions that some versions of calibre update the database which can cause issues when accessing from an older version.

calibre Portable only remembers libraries that are placed inside the portable folder. So while you can open any library located anywhere on your computer with calibre Portable, it will not be remembered on restart unless it is located in the portable folder (i.e. in the same place as the "Calibre Library" folder).

[Epsilon Rose]

Originally Posted by Quoth

An SD card for a Tablet, phone or ereader may not support that Filesystem. Also you need to unmount/mount the SD card to move it between devices and Calibre is broken when you eject / unmount the SD card, if it did have ext4, HPFS (or whatever Apple Mac uses today) or NTFS.

Needing to use compatible file systems is not some weird or insurmountable problem. You just format the card in a format all of the OSs you want to work with understand, and that's pretty easy to do when you just want to work with windows and android.



Also, explain to me how ejecting the SD card is going to break the copy of Calibre stored on the SD card. Please keep in mind that it won't be running while the SD card is not in the computer and while it is running the SD card won't be ejected.