![]() |
#1 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Performance issue with large library
I have a calibre library with just under 60,000 items in it. When adding or editing entries now it is very slow to update the database file and then refresh the screen. I am assuming that the issue is I/O in searching/updating/editing the metatdata.db file which is right at 83 megs in size.
So my question is "is there any way to move the metatdata.db file to my ssd drive while leaving the calibre library in its current location on a larger sata drive." I am running calibre on 64bit ubuntu 11.10 with the os running from a 100 gig pcie ssd drive. All bulk storage is pushed out to standard sata drives. If the issue is I/O performance then moving the database search and updating to the smaller but much much faster ssd should speed stuff up. If that isn't the issue and someone has a suggestion then I am all ears also ![]() thanks |
![]() |
![]() |
![]() |
#2 | |
US Navy, Retired
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Also while doing any editing make sure to hide the tag browser. Hopefully others might have more advice on the subject. Last edited by DoctorOhh; 10-30-2011 at 09:25 AM. |
|
![]() |
![]() |
Advert | |
|
![]() |
#3 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
I have hid the tag browser and don't see a significant improvement and in my installation under linux I don't seem to be able to locate the calibre.bat file either in my home directory or in /opt/calibre where calibre is installed.
|
![]() |
![]() |
![]() |
#4 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
So when I edit metadata what gets changed. Just entries in the database or also in the book directory's on the file system?
|
![]() |
![]() |
![]() |
#5 | |
Well trained by Cats
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 30,217
Karma: 57978778
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
Quote:
![]() any changes to Author or Title affect the file/folder names. all the rest is saved into the DB and is used for conversion, send to device,save (export) to disk AND generating the exportable "Catalog". |
|
![]() |
![]() |
Advert | |
|
![]() |
#6 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
I have found a few posts in the past referencing this issue and that with the calibre.bat file you can change the path of the database. However the files I can find on my linux installation don't seem to match that. What would I need to change on linux to point to a different location for the database?
|
![]() |
![]() |
![]() |
#7 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 4,553
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
The attached file is the Calibre Portable.bat file that was developed for Windows. It is copiously commented, and should provide all the information required to create a linux shell script equivalent.
|
![]() |
![]() |
![]() |
#8 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Based on what I have read about other peoples performance issues I am not being hit to bad. If I'm adding 100 titles or editing tags on them it might take 30 to 60 seconds for calibre to be come responsive again after the files have copied.
I think that my hardware is doing good things for me already. However based on the performance gain related in some other posts the I could probably drop that lag by almost 70 to 80 % by moving the database to my ssd drive. The ssd drive has a throughput of about 400 megs a second while the sata drive is running about 50 megs a second. With an older intel quad core processor and 8 gigs of ram, I don't think there is anything on that side I can speed up. I think it is all I/0 issues. Searching for books in the library now takes a long time also maybe 10 seconds before the search returns results and 20 seconds before the interface becomes responsive again. |
![]() |
![]() |
![]() |
#9 | |
US Navy, Retired
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
This section might help too. Last edited by DoctorOhh; 10-30-2011 at 11:37 AM. |
|
![]() |
![]() |
![]() |
#10 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
I'm not sure I know how to create a script to make this work. I was hoping someone could point me to the file that has this in it and I could just edit a line to make it work.
|
![]() |
![]() |
![]() |
#11 |
US Navy, Retired
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
|
![]() |
![]() |
![]() |
#12 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
ah.. ok i was posting to the one above that and missed your post.. cool. thanks
|
![]() |
![]() |
![]() |
#13 | |
reader
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,975
Karma: 5183568
Join Date: Mar 2006
Location: Mississippi, USA
Device: Kindle 3, Kobo Glo HD
|
Quote:
Code:
export CALIBRE_OVERRIDE_DATABASE_PATH="/home/alan/Calibre Library" |
|
![]() |
![]() |
![]() |
#14 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
Ok I have found a calibre configuration directory under .config in my user directory that has a file called global.py in it that has paths for the database and library.
however the path that was in there does not point to my existing database file though the library path does. I copied moved the database to the root of my home directory and pointed the path to it. However that did not work. after starting calibre it generated a new database in the old location which is the root of the library folder where the books are. |
![]() |
![]() |
![]() |
#15 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
When I try to use the above export path in the /opt/calibre/calibre file
calibre starts and says that the database is corrupted. It isn't as I can remove that line and it will run fine with the database in the root of the library. here is a copy of that file. I have tried the path both as /home/scott and /home/scott/ #!/bin/sh path=`readlink -f $0` base=`dirname $path` lib=$base/lib export LD_LIBRARY_PATH=$lib:$LD_LIBRARY_PATH export CALIBRE_OVERRIDE_DATABASE_PATH="/home/scott" export MAGICK_HOME=$base export MAGICK_CONFIGURE_PATH=$lib/ImageMagick-6.6.7/config export MAGICK_CODER_MODULE_PATH=$lib/ImageMagick-6.6.7/modules-Q16/coders export MAGICK_CODER_FILTER_PATH=$lib/ImageMagick-6.6.7/modules-Q16/filters $base/bin/calibre "$@" |
![]() |
![]() |
![]() |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Best way to get a large Calibre library into the PE library? | Filark | enTourage Archive | 0 | 04-20-2011 10:18 PM |
Networked db/large library a no-go? | concern | Calibre | 3 | 02-03-2010 09:11 PM |
DX performance on large (>100MB) pdf files? | Blackguard | Amazon Kindle | 6 | 06-16-2009 06:57 PM |
Library size and performance | MrFinch | Calibre | 4 | 01-01-2009 04:11 AM |
Performance with large Libraries? | itimpi | Calibre | 3 | 12-14-2008 02:46 PM |