![]() |
#31 |
the rook, bossing Never.
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 12,247
Karma: 89531599
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
|
Close to 70,000 today.
|
![]() |
![]() |
![]() |
#32 | |
the rook, bossing Never.
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 12,247
Karma: 89531599
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
|
Quote:
And having a UPS And hitting Save regularly I found on LO Writer that autosave still runs usually if you somehow managed to freeze the GUI. Just wait the autosave interval you set. |
|
![]() |
![]() |
Advert | |
|
![]() |
#33 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
damn! that would make me want to just sit and cry. 8 gigs is not enough memory in that situation for large operations in calibre much less the rest of the system needing memory also and to have that on an old school hard drive is just brutal. I would bet a author or title search takes several minutes if not more.
|
![]() |
![]() |
![]() |
#34 | |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Quote:
Going from old system in last few months with 1000 mbs nvme and 300,000+ books to new faster cpu and ram and 3500mbs nvme took searches from 4 seconds average to 2 seconds. last comparison is not apple to apple so it is harder to quantify reasons for performance increase than first one that the only difference was going from spin disk to standard ssd. Going from standard 550 mbs ssd to 1000mbs nvme really didn't seem to boost performance a lot when I did that upgrade on my old system. again maybe that had more to do with underlying motherboard as it was 8 years old or so even at that time. I think its about 12 years old now. Actually its the computer im on now. for most desktop tasks it is still super fast. Just limited in IO by older tech. Last edited by audeojude; 02-10-2023 at 07:41 AM. |
|
![]() |
![]() |
![]() |
#35 | |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Quote:
In general I only have about 7 or 8 columns displayed. |
|
![]() |
![]() |
Advert | |
|
![]() |
#36 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
it has been commented about ssd's failing without warning. I have seen that but it is really rare. SSD's do use S.M.A.R.T and it should let you know if you are monitoring it about issues before complete failure.
That being said. If your not backing up data in at least two places other than where your using it your asking to lose it. I use spinney drives to backup my library. One that I keep in a fire proof safe and bring up to date every month or so and one that I do more often. I have a script created just for my library using rsync that I run when I want it to back up. So it is a manual solution and if I get lazy or forgetful I put myself at risk. I have the hardware to build another little server to just use for backups and automate everything but I haven't done it as I don't have enough drive capacity to backup everything that needs it. about 60TB.. for my library yes but for all my other personal and business stuff No. Instead I have a huge stack of labeled hard drives in the 500GB to 4tb range that I break all the data I have up across and I will just drop a drive or drives in an external usb docking station and initiate the rsync job for that drive/datasets. Before anyone points it out I totally agree that that there are much better ways of doing this. However the cost of doing it right due to the size of stuff to be backed up is prohibitive for me. All my data sits on Big raid 10 arrays on a server, made up of 8tb and 10 tb drives ![]() ![]() For most of you with small library's of under 50,000 books, (well any size iibrary really) you should get an external usb hard drive and use the calibre export all books feature as your backup. Then you should get a large flash drive from a reliable manufacture and do it to that also. Use the hard drive for daily or weekly backups and take the flash drive and put it in a fire proof safe or even better off site at a relative or friends house in a small fireproof safe there. Swap it out monthly. The reason I recommend using the export function is it backups up not just the books but also the preferences, plugins and everything else you have customized in your library. Just copying the library directory and database file with rsync like I do for the majority of my backups will not save a lot of that stuff for you. now that you have a off site safe. Take and put copies of all your important documents such as Id, birth certificates, titles etc in it also ![]() Last edited by audeojude; 02-10-2023 at 07:38 AM. |
![]() |
![]() |
![]() |
#37 |
the rook, bossing Never.
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 12,247
Karma: 89531599
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
|
Maybe a SATA SSD uses SMART, but there is no SMART on my NVMe PCIe SSD. The HDD on same PC does have SMART.
|
![]() |
![]() |
![]() |
#38 | |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Quote:
![]() NVME standard in general does not use the SMART system for health reporting but it does have health reporting. My understanding is that the SMART standard is old and really complex (to much so) with a lot of crazy stuff added in over the years hard drive development. The people who developed the NVME standard decided they didn't want to even try and use it because of problems ![]() example https://nvmexpress.org/resources/nvm...architectures/ https://www.pcworld.com/article/5414...free-tool.html https://www.techtarget.com/searchsto...h-of-NVMe-SSDs While smart and sata drives have been around for 3+ decades and is a mature tech, NVME storage is relativly new and doesn't use the sata specification ![]() |
|
![]() |
![]() |
![]() |
#39 |
the rook, bossing Never.
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 12,247
Karma: 89531599
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
|
Much in HDD SMART reporting is irrelevant to SSD.
command line smartctl for Linux only reports on SATA SSD. There is an NVMe command line tool (not installed) but it doesn't work for some or many NVMe SSDs. So I'm mostly right. ![]() Anyway, whatever you are using backups are important. There is always user error, so even if you are living in 1990s and using a magneto-optic disc (which I did use and seem to be near infinite liife and survive a washing machine), there are users. Arghh! That directory has the same name as in another path! RAID certainly doesn't affect needing backups. We sold it in late 1990s as high availability, never as a backup solution. But you know that. Now RAID rebuild times are silly, so now you need two and faster to make a fresh copy than rebuild? Back then we were using 2GB Ultra wide and fast 10,000rpm SCSI. Very noisy! Last edited by Quoth; 02-10-2023 at 09:05 AM. |
![]() |
![]() |
![]() |
#40 |
Well trained by Cats
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 30,334
Karma: 58000000
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
RAID does not protect against real 'melt downs' that can happen. Fire, Water, wild power...
Off site, a goodly distance away and not in a similar location. (I have photos from this month of the families , off grid retreat where the creek that has been dry for the last 3+years, has made the road near impassible to common vehicles (A SxS can make it with detours) for almost 6+miles . ) Plan where carefully. |
![]() |
![]() |
![]() |
#41 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Sorry if I implied that raid was a backup, I did not intend it to sound like that. Raid is a redundancy from single disk failure
![]() Sadly I was able to afford a asustor 6000 series NAS With 10 drive bays that I eventually populated with 6 8tb and 4 10tb drives. years later I still haven't been able to really afford a backup solution other than a huge stack of 500GB to 4tb used sata drives that I can piecemeal put data on for backups. I love my Asustor NAS and their support when in upgrading the size of the raid array it ate the array was stellar. ![]() the sadly comes in that I want to purchase a second one and migrate the data from this one to the new one and use the old one for a backup, but 1200 for the bare NAS and another min 2500 for new larger drives. Ideal would be 3400 dollars for 20tb drives ![]() ![]() Other than the drives I could cheap out and not buy the NAS and just get a used server off of ebay for a roll my own NAS for 3 or 4 hundred dollars that would be way more powerful. The downside of that is your power bill to run the server is high and your power bill to cool the room the server is in is high. Over time you end up spending way more money running it than the cost of a dedicated NAS. My NAS that I am using is specd to at MAX draw 65 watts. So about 1/4 the power usage of most of the 10+ bay servers I could afford on ebay. When I got this NAS and paid the then 900 dollars for it, I was going to do the server route and save 500 to 600 dollars on the cost of it. Before I bought the server I found a server power/cost calculator that you could put how much it cost you per kilowatt and what server you had and get an estimated cost for power to run and cool it. I just about had a heart attack at the estimated annual costs. I paid for the difference in purchase price in the first year of usage on the Asustor unit. About 2500 dollars or more in savings since 2018 when I got it. It's all a bunch of trade off's between what you can afford vs what you want vs what you need vs initial cost vs cost of ownership. |
![]() |
![]() |
![]() |
#42 | |
the rook, bossing Never.
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 12,247
Karma: 89531599
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper11
|
Quote:
I've built decent "servers" using cheap (or thrown out) tower PCs. You only need a "real" server box if you want hot swap HDD trays and Dual hot swap PSUs (swap failed one while running), which needs two separate UPSes! Oh, and serving 100s to 1000s of users rather than a family. Started with NT Server 3.5, then by 2002 switched to Cent OS (basically Red hat), then Debian and now Mint (sort of Debian/Ubuntu). Last Windows server (in parallel with a Linux one for a few years) was Win2000 Server, Server 2003 was too bloated for the old HW. Last edited by Quoth; 02-12-2023 at 05:57 PM. |
|
![]() |
![]() |
![]() |
#43 |
null operator (he/him)
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 20,897
Karma: 27620686
Join Date: Mar 2012
Location: Sydney Australia
Device: none
|
Moderator Notice
This thread has drifted so far off topic I'm thinking I might call Justin & Joe and get them to shoot it down ![]() BR |
![]() |
![]() |
![]() |
#44 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
|
![]() |
![]() |
![]() |
#45 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Somehow I have gotten to where I take a perverse pleasure in kicking this thread once in a while.
So... right now my calibre library is over 350,000 items and running as fast or faster than it ever has. It's now running on a Ryzen 3900 6 cores 12 threads, 32 gigs ram, and a U.2 6.4 TB drive in its own pcie card on the motherboard. because of the new processor I expanded the number of threads/books that can be converted simultaneously and between the speed of the cpu and number of threads allowed it has broken a bottleneck I had when converting 20 or 30 books at the time when pushing them to my kindle. A 30 or 40 minute job is now 1 or 2 at most. Starting the library and searches are very fast given the number of books. Starting is less than 20 seconds and searches are 1 or 2 seconds. Something not mentioned before is the effect of the general OS on performance. Over the last 3 or 4 years the linux kernel has had some very targeted work done on it to remove bottlenecks and increase efficiency and performance overall. I have seen a perceived 20 to 30 % increase in performance. Though my library is on the new ryzen pc and find myself still using my 11 year old 2.5ghz i7. Though it's IO is very abysmal compared to the new system, it's general desktop performance is excellent. Before the kernel enhancements it was still fast but no at the level I'm currently seeing. I think that has also trickled down in general calibre performance. For anyone using calibre with large libraries my recommendations for having good performance in order are. 1. turn the tag browser off for general usage. you will get a 5x performance increase in the responsiveness of the interface. 2. The database and the library both need to live on the fastest hard drive you can afford. If your library is small.. under 10000 items then you can disregard this. However for people like me that are exceeding 100,000 items it is the most crucial aspect of having good performance. PCIE NVME's of one sort or another are recommended. 3. Again for large libraries.. have 16Gig's of ram. Calibre in some operations can use a lot of ram and in general eat up gigs of ram by itself. Smaller libraries it's not a concern don't worry about it. in closing ... don't let these recommendations scare you into not running calibre if you can't afford all the fastest stuff. it will function and run well on old spinney drives, slow cpu's and lower ram. It will just be slower. For the functionality I get out of calibre I would live with that slowness if I had to with a large library. For smaller libraries it will run super fast even on the slowest hardware. last but not least... kick kovid some cash if you use calibre much. I have been using it since 2011 and try and send him 25+ dollars every year or so. Him and the other contributors have done an amazing job. I consider it in my self interest for him to earn enough money to continue updating and extending calibre as I want to be able to use it for many years to come. It's a minuscule amount of money for any individual user but if enough of us do it will make a big difference in his life. I have seen several other opensource projects disappear as it took too much time and effort from the developers regular life over time. I really don't want to one day find that calibre has been abandoned like that. |
![]() |
![]() |
![]() |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Limit on Number of books? | kblechman | Library Management | 1 | 04-02-2011 10:05 AM |
NookLibrary : Limit for number of books? | Kolfe | Barnes & Noble NOOK | 6 | 12-02-2010 06:27 PM |
Limit to number of books in Calibre? | ficbot | Calibre | 1 | 02-26-2010 08:46 AM |
Can calibre handle different books' editions? | italianReader | Calibre | 1 | 01-03-2010 11:13 AM |
Limit to the number of MP3 files? | dhbailey | Sony Reader | 19 | 04-15-2007 04:44 PM |