09-20-2012, 09:15 PM | #1 |
Junior Member
Posts: 3
Karma: 10
Join Date: Sep 2012
Device: Nexus 7
|
Calibre server not updating over NAS
I think I've run into an interesting issue and wanted to see if anyone else can confirm that it is an issue or if I am configuring something improperly.
I am using the folder watch option along with the Calibre server in hopes of having an automated way of adding new books to my collection. My normal setup is to run Calibre in it's own virtual machine with the library located on a NAS mounted over SMB (Windows file sharing). The NAS box is another virtual machine running Linux and Samba. Adding books via the watch folder works fine. Calibre automatically picks up the change and pulls in the book into the library properly. The part that does not work is that the Calibre server does not display the new books unless I restart the server or Calibre itself. What I have discovered is this. If I have both the watch directory and the library on a "local" drive, local in this sense being the same virtual machine, so no NAS in the picture, everything updates properly. When I house the watch folder on the NAS and the library on the local drive, again everything works properly. It's when I put the library on the NAS that there is an issue. The issue with the library sitting on the NAS is that the time stamp on the metadata.db file does NOT get updated properly after a book is added to the library. The timestamp is incorrect from any machine that I mount that same file share. So for example, if I started Calibre at 13:00 the metadata.db file will have the timestamp of 13:00 no matter how many books I add or modify over the course of an entire day. The interesting thing here is that the timestamp on the NAS itself IS correct. So if I login to the Linux NAS box directly and look at the timestamp it reports correctly. However, from any SMB client (Windows for example) the timestamp still says 13:00. I thought let's try to isolate the NAS and see if it's just an SMB issue. My thinking is that perhaps I've configured something incorrectly on the Samba server. So the test was to use a virtual host mount form my virtual machine. This basically presents a local directory to the guest OS as a network share. No real NAS is in play here and everything is running on a Windows 7 box. When I do the same tests over the virtual share, I get the same results. Calibre is running on a Windows XP box (to save on resources) and everything else seems to be running properly. I am suspecting, with no evidence whatsoever, that Calibre is doing something with a file lock such that the SMB protocol is not updating the timestamp properly, thus the Calibre server is not updating properly. Has anyone else encountered this or have any workaround? If not, can anyone help confirm on their machines that this is a reproducible issue? Is there anything I can do to assist in any further debugging? Running Calibre v0.8.69 OS: Windows XP Samba version: 3.6.7 SMBv2 is disabled Library stored on a mapped drive Thanks! Last edited by murkyl; 09-20-2012 at 09:18 PM. Reason: Adding system info |
09-20-2012, 11:38 PM | #2 |
creator of calibre
Posts: 44,409
Karma: 23977332
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
calibre uses the sqlite library to access metadata.db. sqlite requires networked filesystems to implement file locking correctly. Presumably whatever filesystem driver you are using does not. There's no way to fix this in calibre. You either need to change the sqlite locking mechanism or fix the filesystem driver you are using.
|
Advert | |
|
09-23-2012, 02:53 PM | #3 |
Junior Member
Posts: 3
Karma: 10
Join Date: Sep 2012
Device: Nexus 7
|
So I did some additional testing. Everything works fine if I use a Windows 7 machine to host the DB over SMB. It just is not working with my Samba server. I've run a wire shark trace and the SMB protocol is returning back the old time stamp even though the local Linux file system sees the change. I'm still not sure what is causing the issue but I will reply back if I figure this out.
The strange thing is that the same problem occurs when using the VMware host shares. Is anyone else putting the DB on a Linux server shared over SMB? |
09-24-2012, 01:22 AM | #4 |
Junior Member
Posts: 3
Karma: 10
Join Date: Sep 2012
Device: Nexus 7
|
So there is a workaround fortunately! Calibre has the capability to have the metadata database in a different directory from the library itself. You need to set it with the full path including the name of the database file in an environment variable named:
e.g. C:\Some\Path\metadata.db This is documented here: http://manual.calibre-ebook.com/customize.html This link here in the forums also helped: https://www.mobileread.com/forums/sho...d.php?t=190436 It turns out there is some incompatibility between SQLite and Samba. You can find a ton of posts about this if you search on 'sqlite cifs'. It looks like it is a form of file locking which is causing the problem. A full Microsoft client and server do no seem to have this problem. It seems to only occur if either the client or server is running Samba. I've used the environment variable to move the DB and now the Calibre server updates properly when new books are added. The last step is to backup the metadata DB files now that they are no longer on the NAS. Hope this helps someone else in the future. |
02-09-2013, 01:53 PM | #5 | |
Junior Member
Posts: 4
Karma: 10
Join Date: Jan 2012
Device: Kindle Keyboard 3G
|
Quote:
I edited the calibre-portable.bat file to set the ebook location as follows: Code:
IF EXIST N:\eBooks ( SET CALIBRE_LIBRARY_DIRECTORY=N:\eBooks ECHO LIBRARY FILES: N:\eBooks ) Code:
IF EXIST "C:\Documents and Settings\Administrator\Calibre Library\metadata.db" ( IF NOT "%CALIBRE_LIBRARY_DIRECTORY%" == "C:\Documents and Settings\Administrator\Calibre Library" ( SET CALIBRE_OVERRIDE_DATABASE_PATH="C:\Documents and Settings\Administrator\Calibre Library\metadata.db" ECHO DATABASE: C:\Documents and Settings\Administrator\Calibre Library\metadata.db ECHO ' ECHO ***CAUTION*** Library Switching will be disabled ECHO ' ) ) However, when I launch the calibre-portable.bat file, I get the following error: Code:
calibre, version 0.9.17 Corrupted database: The library database at N:\eBooks appears to be corrupted. Do you want calibre to try and rebuild it automatically? The rebuild may not be completely successful. If you say No, a new empty calibre library will be created. Traceback (most recent call last): File "site-packages\calibre\gui2\main.py", line 228, in initialize_db File "site-packages\calibre\library\database2.py", line 166, in __init__ File "site-packages\calibre\library\database2.py", line 114, in connect File "site-packages\calibre\library\sqlite.py", line 366, in connect DatabaseException: unable to open database file RemoteTraceback (most recent call last): File "site-packages\calibre\library\sqlite.py", line 248, in run File "site-packages\calibre\library\sqlite.py", line 225, in connect OperationalError: unable to open database file Am I missing a step? Thanks for the help! |
|
Advert | |
|
02-09-2013, 06:40 PM | #6 |
Wizard
Posts: 4,553
Karma: 950151
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
|
It is definitely possible to run with that sort of setup - I do it all the time.
To help yourself identify what is wrong, it is probably worth removing the 'echo off' at the start of the batch file. If you then start it from within a command window you can see the various lies of the batch file executing which may help? If you still do not get anywhere by tomorrow leave another post, and I can post my batch file as a comparison (I am not at home until tomorrow evening). |
02-10-2013, 12:23 AM | #7 |
Junior Member
Posts: 4
Karma: 10
Join Date: Jan 2012
Device: Kindle Keyboard 3G
|
Thanks for your reply. I ended up getting everything working. I made an actual environment variable on my system instead of just using the calibre-portable.bat. Doing this worked right away.
It is taking several minutes to load my library though. It seems faster than before, but still takes a while. Not complaining, but I am wondering if this is fairly normal. Anything I can do to speed it up? I have disabled several columns in the main display window which helped loading times. |
Tags |
server update nas smb |
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Calibre Server Not Updating | Fuller | Calibre | 1 | 11-21-2011 01:36 PM |
Calibre Content server and Iomega NAS | subivan | Devices | 9 | 08-12-2011 05:26 PM |
Calibre-server not updating | skells | Calibre | 16 | 07-22-2011 04:38 PM |
Headless server on NAS readonly problem | artifact | Related Tools | 1 | 06-28-2011 09:34 PM |
Library on NAS server? | bous | Calibre | 5 | 11-26-2009 05:03 AM |