Biggest database Calibre can handle?

[manawydan]

Since I have problems with Calibre ever since my database grew bigger (>30k files) I wonder what is the biggest database Calibre is able to handle?

Of course it can also be a matter of hardware/OS.
But I have got a newer PC and there are still problems.
The most obvious when starting Calibre. It takes 10-15 minutes (or longer - I do no longer check) till the programme is useable.

Well, this should not be a question for a solution or tipps but I am curious how big the average database is.

[Adoby]

I expect that calibre could handle several 100K entries without serious problems, if you use some care.

Things that can slow down calibre a lot is if you use custom columns, based on other columns. Or things like rules to color fields. Or if you have a formats column that is based on checking the formats of the actual ebooks. Or if you have the tag browser open if not needed.

By default I believe that the formats displayed now use information stored in the database, about the books, and not actually check the book files when calibre is started. That can be very slow. But if you have been using calibre for a while your installation may use the old method.

You can read more about this in this now a bit old thread:

https://www.mobileread.com/forums/sho....php?p=2244745

Since that thread was started calibre has gotten a new database backend that has made calibre much faster in many cases. The problems that I used to have, and that caused me to start that thread, are now mostly gone. But I still don't have any computed custom columns and I use approximate format and I have the database on a SSD drive using symlinks and temp in RAM...

Also you may be able to split your calibre library into two or three separate libraries. That might also help.

For instance Fiction and Nonfiction. And perhaps Unsorted for new/incoming books that you haven't fixed metadata for, or need to fix other things for.

* Remove any columns you don't actually use, especially if they depend on other columns.
* Check to make sure that the formats column is based on approximate format and not actual format.
* Hide the tag browser when you don't use it.
* Add more memory to the computer.

[BetterRed]

Adoby

I recently had start times go from a few seconds to a few minutes if I started on a specific library, and switching to the library also became slow. It started when the library reached about 6,000 books. The library also became sluggish when scrolling the library list etc.

However, I have another library with 32,000 books, starting or switching to it takes just a few seconds. So it was obviously something specific to that 6K book library.

One of the custom columns on the 6K book library was derived via a union of four other columns, each of which contained a list of names. When I removed that derived column the start/switch times went back to a few seconds. After rearranging the order of columns in the library list, and in Book Details I was able to arrive at a layout where I didn't lose much by not having the derived column.

It was interesting that the presence of the derived column had no apparent effect until the library went through a tipping point. I played around and found that the start times started to rise at about 5600 books and became progressively longer so that at 6100 books the start time was about 5-6 minutes

I suspect the tipping point is a function of the other factors mentioned by Adoby as well as the nature of the derived column that creates the delays. That library has two other derived columns that use simpler templates, it also has several column icons - removing them makes no appreciable difference to start/switch times but they probably have there own tipping points.

BR

[manawydan]

Originally Posted by Adoby

Things that can slow down calibre a lot is if you use custom columns, based on other columns. Or things like rules to color fields. Or if you have a formats column that is based on checking the formats of the actual ebooks. Or if you have the tag browser open if not needed.

I have custom columns but not based on others. No tag browser, no color rules ..
I have never been able to find out why I have these problems and by now I just live with it.

I will check the thread you linked.
Did I get you right, instead of simply upgrading you suggest I should keep the library (the actual book files folder), uninstall and/or install calibre as new?

[Adoby]

I didn't say anything about upgrading. I assumed that you are using the latest version of calibre. If not, just upgrade. Nothing special.

And delete columns you don't use. Especially the formats column if it isn't based on approximate formats.

And check for columns that are not displayed. Delete them if you don't use them. I remember cases when users insisted that they didn't have any formats column, that checked the files on disk, but it turned out there was one after all...

If you want to see if a fresh library will improve performance, create a new library and copy the contents of the old library to the new. You can do this from within the calibre GUI.

Also, if you use an old install of XP that is beginning to get shaky and slow, you might consider reinstalling the OS or upgrading.

[BetterRed]

I wonder if its related to this ==>> https://www.mobileread.com/forums/sho...d.php?t=193686.

The cause of the OPs problem starts to reveal itself at post 33. The solution is at posts 50/51. And if it solves your problem send some karma to PeterT.

Otherwise, as a quick test you could try creating another library with no customisations, copy say 20 books from existing library but DON'T create missing custom columns - restart calibre

If calibre is still slow to start then the problem is probably NOT related to the 30K+ library. What happens if you start calibre in Windows safe mode?

BR

[speakingtohe]

I have a lot of books and a lot of columns. 5 are columns built on other columns. Calibre starts in seconds even on my older laptop with library on older external drive.

Not much help I know, but it should work better. I would definitely try upgrading.

Helen

[Adoby]

If it is an old library, then the formats might be updated by checking the actual book files every time calibre is started. That would cause serious lag when starting, especially if sorting on the formats.

Easy to check and fix. Just delete the formats column. Easy to add back later if needed.

When you have problems with calibre, it usually is a good idea to upgrade and see if the problems have been fixed in newer versions. New versions of calibre usually improve the quality of the program. That is not true for all software...