Are too many single-use tags (in tags-like field(s)) bad for Calibre's performance?

[01100001]

Hello all, again!

I'm using a custom column called Filenames (#filenames) of type Comma separared text, like tags, shown in the Tag browser.

What this is means is that as I add more and more books, there will be a ton of single-use tags in this custom column/field. (And by single-use tags I mean, that all tags will be used just once, besides there may be multiple per book, so they'll add up pretty quickly.)

Question: Since calibre offers to auto-fill these fields, apart from showing the whole lot in the Item editor and the Tag browser (where I disabled the category), I would like to know if this will be a problem for Calibre's performance in the long-term.

I also ask because, for another library/project I intend to import a huge database (via CSV format) of hundreds of thousands of books. But this is also a concern for my personal library (5k books).

Why do I use a tags-like field for filenames?

I believe I explained it in good detail here.

[BetterRed]

Moderator Notice

I can't be bothered spending time trying to comprehend posts that have images that violate Mobileread Guidelines, specifically #9. Please use the MR attachment facility, then you can post thumbnails. Hiding large images in spoiler tags is not acceptable.

BR

[01100001]

Fixed. Another user made me aware of this later on, but I didn't go back and fix my older posts. Thank you.

[chaley]

If there is a problem then it will be startup speed & memory. Calibre reads the DB into memory when it starts. The bigger the DB the more time this takes and the more memory is required.

I use the word "tags" here. Custom tags-like columns are the same.

Tags are linked to books using bog-standard N-M relational links. By that I mean the structure "books <- tag_links -> tags". Given a book (book id) it is one (indexed) query to find all the tag ids referenced by that book. From there it is a simple (indexed) query to get the tag values. Given a tag id it is one (indexed) query to find the book ids that reference that tag.

Given the above, I don't see significant problems with having lots of tags referenced by single books, where any given book references a reasonable number of such tags (say, less than 100). Where I would expect problems is if books each reference hundreds of thousands of tags, unique or not. This problematic case requires walking those tags for each book to display the data.

Bottom line: if you have lots of memory then I doubt you will notice anything. If you don't then you might get into memory swapping/paging, which will dramatically slow things down.

[01100001]

That makes sense. Mine is a much simpler case. 5 filenames per book max., and just 1 on most, like 99% of the time.

Thank you very much for the detailed explanation!

Conclusion: It won't be a problem for me, at all.

[BetterRed]

@01100001 - I'm still at a loss to understand why you want a custom column containing the file paths of a book's format files - which is effectively what the Formats column provides.

BR

[theducks]

There is no 'Formats' column.
OTOH the info (what formats) shows in a number of places: MDE, Info dialog: Details pane.
The Path is relative to the Library. O(pen) takes you to the location (folder)of the current book.
Save Single Format to disk will give you a choice without you needing to know where Calibre keeps the book file

[BetterRed]

Originally Posted by theducks

There is no 'Formats' column.

Easy enough to create one with the approximate_formats template function.

But why would you create a custom column using this {:'formats_paths()'} to get this.

DOCX:D:\Calibre\Libraries\Test\Anat Admati\The Bankers' New Clothes (159)\The Bankers' New Clothes - Anat Admati.docx, EPUB:D:\Calibre\Libraries\Test\Anat Admati\The Bankers' New Clothes (159)\The Bankers' New Clothes - Anat Admati.epub, PDF:D:\Calibre\Libraries\Test\Anat Admati\The Bankers' New Clothes (159)\The Bankers' New Clothes - Anat Admati.pdf

I can think of a couple of obscure reasons, but the OP hasn't given any.

BR

[01100001]

Originally Posted by BetterRed

@01100001 - I'm still at a loss to understand why you want a custom column containing the file paths of a book's format files - which is effectively what the Formats column provides.

Ha! I don't store my ebook files in Calibre. I use it to catalog my reading interests and their metadata only (i.e. I add empty books).

I prefer it this way for three reasons (off the top of my head): (1) Faster backups (2) Files named and organized how I want them on the filesystem (3) Don't have to be worried about storage space running out on my PC. Ebooks are backed-up on multiple external disks.

[BetterRed]

Have you considered putting file:\\ uris into a long text column via the insert link tool. See attachments

Downside: if you move or change a file name or location the links won't work any more.

BTW I could take issue with all your reasons for not storing content in the library.

BR

[01100001]

Yes, of course, and then rejected the idea. My PCs are a Surface Pro 7 and a MacBook Pro with 128 GB SSDs. I can spare at most 10 GB for books, which isn't enough for my entire collection. So I store them away in a HDD (and replicate 'em on another). That's also why I only store filenames. Paths tend to change. (EDIT: But I want my catalog with me always.)

The tags-like Filenames (#filenames) field allows me to do a bunch of things, like (1) generate file counts (no. of ebooks per title; sometimes there are multiple versions of a title in the same format); (2) auto-generate paths into a custom-column, etc. I don't need these now, but whatever I may need in the future, this field has me covered, at least better than any other, I think.

BTW I could take issue with all your reasons for not storing content in the library.

I'd like to see you try. I don't mind someone changing my mind.

PS: I read somewhere that a thin-client (Calibre app on PC connects to Calibre running remotely) is in Kovid's plans. I don't know if it still is, but if it ever comes to fruition, I may feel comfortable letting Calibre hande everything.