Is there a limit to the number of books Calibre can handle?

[AussieTupp]

Hi ... my first post, although I do pop in and browse around occasionally and have found a lot of incredibly useful information and snippets here.

(Aside: I must follow up how to put metadata on the first page of a book!!!!! What a ripper! )

What I am wondering though is if there is an upper limit to the number of books Calibre can handle. I vaguely recollect somewhere in the early days of looking around here that someone mentioned 50,000 (Fifty THOUSAND? Yikes!) but I've been unable to find that information again.

I don't have anywhere near that but cycle a lot through my Kindle 3, which I love to bits and take everywhere with me.

Thanks for any help ...

[DoctorOhh]

Originally Posted by AussieTupp

Hi ... my first post, although I do pop in and browse around occasionally and have found a lot of incredibly useful information and snippets here.

I don't know the practical limit but here is a recent post from a person that has 200,000 books in their library.

[AussieTupp]

Thanks for the prompt answer dwanthny. Highly unlikely I'll ever have anything remotely like that number of books ... I have trouble reading all the ones I have!

[DoctorOhh]

Originally Posted by AussieTupp

Thanks for the prompt answer dwanthny.

You're Welcome. I currently have 10,000 books in my library with no performance hit whatsoever.

Good Reading.

[audeojude]

This is an old thread but I though people would be interested. I have almost 230,000 books in my library and it runs like a rock. With this many books and as many directories (more than 40,000) that are in the root of the calibre library directory performance was big issue starting at about 100,000 books. I moved the library to it's own dedicated ssd on a sata3 interface and now the largest lag is 4 or 5 seconds instead of 30 seconds to a minute on search queries. It also helps that it is a 2.5 ghz i7 with 32 gigs or ram. I just checked and on a linux system with calibre running since a reboot yesterday it is using 5 gig's of memory on its own.

[josykatz]

Thank you. I'm glad to know that. I have a ton of books and they are all over the place. I'm now trying to consolidate them to one folder so I can have calibre search and put everything away correctly

[ownedbycats]

I know this is an old thread, but I was curious about it too, mostly the hard limit of books the Calibre database can hold.

Calibre uses a sqlite database, and according to the SQLite documentation, the theoretical maximum number of rows in a table is 2*64 (no superscript?), which is....a very big number. Though you'll reach the 140 terabyte database size limit long before that number.

[audeojude]

Originally Posted by ownedbycats

I know this is an old thread, but I was curious about it too, mostly the hard limit of books the Calibre database can hold.

Calibre uses a sqlite database, and according to the SQLite documentation, the theoretical maximum number of rows in a table is 2*64 (no superscript?), which is....a very big number. Though you'll reach the 140 terabyte database size limit long before that number.

Havent run into it yet.. Over 60,000 directories and 260000 items in library. Now running on a 2tb intel M3 drive at about 1000 mb a second. To be honest I don't really see much performance increase over the old 1tb samsung ssd running at 500ish MB a second in searches etc.. Though that might be due to System IO bottlenecks either hardware or OS. I don't feel as if I am getting the IO performance that my underlying drives should be getting. Average author search in my library takes 4 to 8 seconds to come back. A bulk convert of a bunch of books seems to average 2 to 15 seconds per book. Though doing it as part of syncing some books to my kindle takes much much longer. If I convert and then sync it a much shorter process. Starting calibre after I boot system takes a minute to a minute and a half before it's ready to use. I think this lag is mostly due to the initial look at the directory the library is in because of the vast amount of directories in it. This is an underlying OS issues not a calibre issue. I use the built in server which works well, I am not sure if it affects start up time if it is enabled. I am not sure on searches why it takes as long as it does to come back with a reply even if there are 260000 books in library. I know I deal with big databases that make that look really really small that don't have that kind of lag on much slower infrastructure. I have just assumed that it has to do with the type of database used that is embedded in calibre. However performance is good enough that I am not frustrated using it. And over the years it has gotten incrementally and significantly better performance wise over time as it has been improved. At one time with less than half the books in it searches might take 20ish seconds compared to now.

My current system though fast is 10 years old now. It's fast enough that I haven't been able to justify the 1000 dollars or so to upgrade to one of the new AMD's systems with more cores and lots of PCIE lanes. Maybe late this year or next year.

Remember to every once in a while kick a donation through paypal to the developers. This is an awesome program that I have used since pre 2012. I try to send some money every year or two to them. Same with other open source programs I use on a regular basis. I would hate to see Calibre disappear like so many other programs I have used in the past.

Update 1/2023 see new post from me below. significant performance increase since this with new hardware and 30% ish larger library

[ownedbycats]

I figure if my Calibre superlibrary ever gets big enough that it starts lagging, I'll just split it into a few smaller libraries. That'll be a while yet as I've got less than 4000 books in there so far.

[theducks]

Originally Posted by ownedbycats

I figure if my Calibre superlibrary ever gets big enough that it starts lagging, I'll just split it into a few smaller libraries. That'll be a while yet as I've got less than 4000 books in there so far.

Remember: It is not about the size of the books, but the size of the DB entries.
Extra columns, large comments, all make the metadata.db bulky.

Bulky is what can slow down normal activities.

OTOH Large books, slow down conversions and file transfers.

[ownedbycats]

Yeah, the one thing I'm a little worried about is the fanficfare metadata long-text column, but so far it's proven useful enough to keep.

[audeojude]

I tend to pull descriptions from amazon so most everything has a paragraph or two in the comments field. Books size averages between .5 to 1.5 mb. 200 pages to 600 pages. Metadata.db file is 545 MB in size.

[audeojude]

with only 4000 books in your library you are a long way from needing to worry at all. I have about 64x that many entries.

[ownedbycats]

Thankfully the filesize of the book itself doesn't affect the database much, which is a good thing because I have some rather bulky files.

My metadata.db is about 15 MB in size, although I distinctly remember it being around 2-3 MB before I started using FanFicFare's long-text metadata column. Though after doing a lot of metadata edits and vacuuming the database, it tends to shave off a few kilobytes.

[georgestancliffe]

And I thought my E-Book collection was BIG. I have about 7,000 E-books in my laptop. Most of them were downloaded from Gutenberg.org, for free. If I read one book each day, it would take me about 20 years to read them all!

I'm glad to hear that the CALIBRE E-book reader can handle over 250,000-book super-libraries!