Is there a limit to the number of books Calibre can handle?

[Quoth]

Close to 70,000 today.

[Quoth]

Originally Posted by DoctorOhh

You say this as if performing backups isn't essential with all drives.

Of course it is. Just replaced HDD and OS today on the PC with the SCSI scanner and drive to make CDs into MP3s, but no user data lost, though almost all of the old HDD is still readable. SDDs tend to completely fail. HDDs also have SMART utility.

And having a UPS
And hitting Save regularly

I found on LO Writer that autosave still runs usually if you somehow managed to freeze the GUI. Just wait the autosave interval you set.

[audeojude]

Originally Posted by DNSB

The most I remember one poster mentioning was having close to 1,000,000 items in a calibre library. The main complaints were that startup times were longer and most operations were slower. The computer had 8GB of RAM and a spinning rust hard drive.

damn! that would make me want to just sit and cry. 8 gigs is not enough memory in that situation for large operations in calibre much less the rest of the system needing memory also and to have that on an old school hard drive is just brutal. I would bet a author or title search takes several minutes if not more.

[audeojude]

Originally Posted by mbovenka

The 'spinning rust' bit actually matters less than you'd think. Calibre loads the metadata db into RAM at startup (so startup would be a bit slower), and searches and such are done on the in-memory copy. So only bulk updates would be slower, and you're not likely to do all that many of those.

RAM and CPU are more important, and 8GB RAM probably is pushing it with 1M database entries.

Theoretically I would agree with you but in practice just going from sata hard drive to ssd with all else held equal reduced start up time and search time drastically. Searches that were in the average 30 seconds when I did the initial upgrade from spin disk to ssd went to 4 seconds average.

Going from old system in last few months with 1000 mbs nvme and 300,000+ books to new faster cpu and ram and 3500mbs nvme took searches from 4 seconds average to 2 seconds. last comparison is not apple to apple so it is harder to quantify reasons for performance increase than first one that the only difference was going from spin disk to standard ssd. Going from standard 550 mbs ssd to 1000mbs nvme really didn't seem to boost performance a lot when I did that upgrade on my old system. again maybe that had more to do with underlying motherboard as it was 8 years old or so even at that time. I think its about 12 years old now. Actually its the computer im on now. for most desktop tasks it is still super fast. Just limited in IO by older tech.

[audeojude]

Originally Posted by DNSB

Most people seemed to agree with you. Suggesting an upgrade in memory and CPU and upgrading to a NVME SSD was the consensus. Reducing the number of custom columns wa also popular.

I do remember I used to have the tag browser displayed all the time. Closing that increased general performance a bit. That would come under reducing number of columns .
In general I only have about 7 or 8 columns displayed.

[audeojude]

it has been commented about ssd's failing without warning. I have seen that but it is really rare. SSD's do use S.M.A.R.T and it should let you know if you are monitoring it about issues before complete failure.

That being said. If your not backing up data in at least two places other than where your using it your asking to lose it. I use spinney drives to backup my library. One that I keep in a fire proof safe and bring up to date every month or so and one that I do more often. I have a script created just for my library using rsync that I run when I want it to back up. So it is a manual solution and if I get lazy or forgetful I put myself at risk. I have the hardware to build another little server to just use for backups and automate everything but I haven't done it as I don't have enough drive capacity to backup everything that needs it. about 60TB.. for my library yes but for all my other personal and business stuff No. Instead I have a huge stack of labeled hard drives in the 500GB to 4tb range that I break all the data I have up across and I will just drop a drive or drives in an external usb docking station and initiate the rsync job for that drive/datasets.

Before anyone points it out I totally agree that that there are much better ways of doing this. However the cost of doing it right due to the size of stuff to be backed up is prohibitive for me. All my data sits on Big raid 10 arrays on a server, made up of 8tb and 10 tb drives other than the library that needs it's own nvme for performance. So for most of it a bare minimum of redundancy before the single backup to external drives. Library gets a little bit more in the way of backup. When I win the lottery I will have multiple backups with my own personal off site cloud backup also

For most of you with small library's of under 50,000 books, (well any size iibrary really) you should get an external usb hard drive and use the calibre export all books feature as your backup. Then you should get a large flash drive from a reliable manufacture and do it to that also. Use the hard drive for daily or weekly backups and take the flash drive and put it in a fire proof safe or even better off site at a relative or friends house in a small fireproof safe there. Swap it out monthly. The reason I recommend using the export function is it backups up not just the books but also the preferences, plugins and everything else you have customized in your library. Just copying the library directory and database file with rsync like I do for the majority of my backups will not save a lot of that stuff for you.

now that you have a off site safe. Take and put copies of all your important documents such as Id, birth certificates, titles etc in it also

[Quoth]

Maybe a SATA SSD uses SMART, but there is no SMART on my NVMe PCIe SSD. The HDD on same PC does have SMART.

[audeojude]

Originally Posted by Quoth

Maybe a SATA SSD uses SMART, but there is no SMART on my NVMe PCIe SSD. The HDD on same PC does have SMART.

ok, this is a bit of a mess as you are both right and wrong

NVME standard in general does not use the SMART system for health reporting but it does have health reporting. My understanding is that the SMART standard is old and really complex (to much so) with a lot of crazy stuff added in over the years hard drive development. The people who developed the NVME standard decided they didn't want to even try and use it because of problems in that it would make them build in stuff they didn't want into the NVME drives. (scratch head, my understanding on this and why is minimal) However they did build in all the health reporting stuff that you would need. I'm assuming based on below that some of the smart software is being modified to read some NVME information now.

example
https://nvmexpress.org/resources/nvm...architectures/

https://www.pcworld.com/article/5414...free-tool.html

https://www.techtarget.com/searchsto...h-of-NVMe-SSDs

While smart and sata drives have been around for 3+ decades and is a mature tech, NVME storage is relativly new and doesn't use the sata specification so it's not as cohesive at a user level yet but it is there.

[Quoth]

Much in HDD SMART reporting is irrelevant to SSD.
command line smartctl for Linux only reports on SATA SSD.
There is an NVMe command line tool (not installed) but it doesn't work for some or many NVMe SSDs.

So I'm mostly right.

Anyway, whatever you are using backups are important. There is always user error, so even if you are living in 1990s and using a magneto-optic disc (which I did use and seem to be near infinite liife and survive a washing machine), there are users. Arghh! That directory has the same name as in another path!

RAID certainly doesn't affect needing backups. We sold it in late 1990s as high availability, never as a backup solution. But you know that.
Now RAID rebuild times are silly, so now you need two and faster to make a fresh copy than rebuild? Back then we were using 2GB Ultra wide and fast 10,000rpm SCSI. Very noisy!

[theducks]

RAID does not protect against real 'melt downs' that can happen. Fire, Water, wild power...
Off site, a goodly distance away and not in a similar location.
(I have photos from this month of the families , off grid retreat where the creek that has been dry for the last 3+years, has made the road near impassible to common vehicles (A SxS can make it with detours) for almost 6+miles . ) Plan where carefully.

[audeojude]

Sorry if I implied that raid was a backup, I did not intend it to sound like that. Raid is a redundancy from single disk failure I was just specifying where most of my data resides.

Sadly I was able to afford a asustor 6000 series NAS With 10 drive bays that I eventually populated with 6 8tb and 4 10tb drives. years later I still haven't been able to really afford a backup solution other than a huge stack of 500GB to 4tb used sata drives that I can piecemeal put data on for backups.

I love my Asustor NAS and their support when in upgrading the size of the raid array it ate the array was stellar. they were able to fix it with no loss of data.. took 4 days of re-initializing the array but they fixed it. Best I could tell there was a bug in the firmware that affected the exact method I tried to use in expanding the size of the array. Other than that one issue it has run like a rock for years now.

the sadly comes in that I want to purchase a second one and migrate the data from this one to the new one and use the old one for a backup, but 1200 for the bare NAS and another min 2500 for new larger drives. Ideal would be 3400 dollars for 20tb drives and then have the same issue going forward of no backups after I exceed the old units storage capacity unless i spend another 1500 dollars for 4 20 tb drives to use as external backup. Having lots of data cost lots of money to maintain it correctly

Other than the drives I could cheap out and not buy the NAS and just get a used server off of ebay for a roll my own NAS for 3 or 4 hundred dollars that would be way more powerful. The downside of that is your power bill to run the server is high and your power bill to cool the room the server is in is high. Over time you end up spending way more money running it than the cost of a dedicated NAS. My NAS that I am using is specd to at MAX draw 65 watts. So about 1/4 the power usage of most of the 10+ bay servers I could afford on ebay. When I got this NAS and paid the then 900 dollars for it, I was going to do the server route and save 500 to 600 dollars on the cost of it. Before I bought the server I found a server power/cost calculator that you could put how much it cost you per kilowatt and what server you had and get an estimated cost for power to run and cool it. I just about had a heart attack at the estimated annual costs. I paid for the difference in purchase price in the first year of usage on the Asustor unit. About 2500 dollars or more in savings since 2018 when I got it.

It's all a bunch of trade off's between what you can afford vs what you want vs what you need vs initial cost vs cost of ownership.

[Quoth]

Originally Posted by audeojude

Sorry if I implied that raid was a backup, I did not intend it to sound like that. .

No, you didn't, but we had to educate customers in 1990s that it was a High Availability solution rather than a backup.

I've built decent "servers" using cheap (or thrown out) tower PCs. You only need a "real" server box if you want hot swap HDD trays and Dual hot swap PSUs (swap failed one while running), which needs two separate UPSes! Oh, and serving 100s to 1000s of users rather than a family.

Started with NT Server 3.5, then by 2002 switched to Cent OS (basically Red hat), then Debian and now Mint (sort of Debian/Ubuntu). Last Windows server (in parallel with a Linux one for a few years) was Win2000 Server, Server 2003 was too bloated for the old HW.

[BetterRed]

Moderator Notice

This thread has drifted so far off topic I'm thinking I might call Justin & Joe and get them to shoot it down

BR

[audeojude]

Originally Posted by BetterRed

This thread has drifted so far off topic I'm thinking I might call Justin & Joe and get them to shoot it down

BR

Lol Your no fun!

[audeojude]

Somehow I have gotten to where I take a perverse pleasure in kicking this thread once in a while.

So...

right now my calibre library is over 350,000 items and running as fast or faster than it ever has.

It's now running on a Ryzen 3900 6 cores 12 threads, 32 gigs ram, and a U.2 6.4 TB drive in its own pcie card on the motherboard.

because of the new processor I expanded the number of threads/books that can be converted simultaneously and between the speed of the cpu and number of threads allowed it has broken a bottleneck I had when converting 20 or 30 books at the time when pushing them to my kindle. A 30 or 40 minute job is now 1 or 2 at most.

Starting the library and searches are very fast given the number of books. Starting is less than 20 seconds and searches are 1 or 2 seconds.

Something not mentioned before is the effect of the general OS on performance. Over the last 3 or 4 years the linux kernel has had some very targeted work done on it to remove bottlenecks and increase efficiency and performance overall. I have seen a perceived 20 to 30 % increase in performance. Though my library is on the new ryzen pc and find myself still using my 11 year old 2.5ghz i7. Though it's IO is very abysmal compared to the new system, it's general desktop performance is excellent. Before the kernel enhancements it was still fast but no at the level I'm currently seeing. I think that has also trickled down in general calibre performance.

For anyone using calibre with large libraries my recommendations for having good performance in order are.

1. turn the tag browser off for general usage. you will get a 5x performance increase in the responsiveness of the interface.

2. The database and the library both need to live on the fastest hard drive you can afford. If your library is small.. under 10000 items then you can disregard this. However for people like me that are exceeding 100,000 items it is the most crucial aspect of having good performance. PCIE NVME's of one sort or another are recommended.

3. Again for large libraries.. have 16Gig's of ram. Calibre in some operations can use a lot of ram and in general eat up gigs of ram by itself. Smaller libraries it's not a concern don't worry about it.


in closing ... don't let these recommendations scare you into not running calibre if you can't afford all the fastest stuff. it will function and run well on old spinney drives, slow cpu's and lower ram. It will just be slower. For the functionality I get out of calibre I would live with that slowness if I had to with a large library. For smaller libraries it will run super fast even on the slowest hardware.

last but not least... kick kovid some cash if you use calibre much. I have been using it since 2011 and try and send him 25+ dollars every year or so. Him and the other contributors have done an amazing job. I consider it in my self interest for him to earn enough money to continue updating and extending calibre as I want to be able to use it for many years to come. It's a minuscule amount of money for any individual user but if enough of us do it will make a big difference in his life. I have seen several other opensource projects disappear as it took too much time and effort from the developers regular life over time. I really don't want to one day find that calibre has been abandoned like that.