Using Calibre as a client for another Calibre instance?

[toddos]

I love Calibre and currently have it running on my Windows Home Server. I can manage it using remote desktop and download books on my iPhone using Stanza without having to connect the iPhone to the physical machine running Calibre. One little problem, though. I purchased a Sony ebook reader through yesterday's woot deal. It should arrive in the next couple of weeks so I'm getting prepared, and unfortunately the Sony doesn't have wifi to download books.

I realize Calibre can sync to the Sony easily enough if it's connected to the computer, but I rarely physically access my WHS. I'd like to keep my library consolidated on the WHS, and I keep Calibre runnin 24/7 so I can download on the iphone whenever I like, but now I also need a convenient way to get books onto the Sony. Ideally, I'd be able to run an instance of Calibre on my laptop that connects to the instance of Calibre running on the WHS for library access rather than recreating my library locally. Is this possible to do? I'd even be happy with a simple read-only view of the library that can just sync and not do any adding or editing. The one thing I can't do is share the database file from the server instance because Calibre locks the file as long as it's running and the server instance is always running.

Otherwise, I'm afraid that my only option may be to rely on Remote Desktop's ability to share USB devices with remote computers. Unfortunately that's often been buggy for me in the past and WHS is based on comparatively ancient technology (Windows Server 2003) and may not fully support device sharing scenarios.

Help! I really don't want to have to physically access my WHS all the

[Manichean]

I've not tried this, but maybe you should have a look at using Calibre2OPDS together with a Calibre library on a shared network drive. I'm thinking you should be able to run Calibre on your workstation with the library location being on the shared drive (as in "on your server"), and the server running, well, the server I linked to above.

[solomon]

Take a look at the sticky thread about Calibre-server. Here in Post #125 I asked about running *JUST* calibre-server on WHS and the GUI on a client machine. KovidGoyal says it should work, since the server process will be Read-Only and the client will be Read-Write.

I think that will work for you as well - by pointing your GUI Calibre to the WHS directory for its library.

If you are *VERY* careful, you might be able to run the GUI on multiple machines - but like Ghostbusters, NEVER CROSS THE STREAMS I would NOT recommend this myself though - as you must be 100% certain that only one machine has Calibre running at any time.

(There's another thread here which goes through that multi-client idea in excruciating detail.)

HTH,
Richard

[toddos]

calibre-server is just the web front-end, right? That might work. The only reason I have calibre running 24/7 is for stanza access. I could do gui management stuff from my laptop pointing to the server shares for ebook and database storage. I'll play with that tonight and see if it satisfies my needs.

It sure would be nice if calibre supported mysql, postgresql, sql server express, or similar as a database target, though. That would remove database locking/corruption risks from multi-client scenarios, though you'd still have to deal with stale data, actual ebook file access, etc.

[solomon]

Originally Posted by toddos

calibre-server is just the web front-end, right? That might work. The only reason I have calibre running 24/7 is for stanza access. I could do gui management stuff from my laptop pointing to the server shares for ebook and database storage. I'll play with that tonight and see if it satisfies my needs.

Yes, and ditto - though now I want it for Trook too!

Please do post back - either here or in the other thread. I probably won't have a chance to try it this weekend

I presume we'll still want SRVANY to get calibre-serve running as a service on the WHS box, but that's more of a detail.

Richard

[toddos]

I just got this all set up and working. Warning! Before you do anything, make a backup of all of the data in Calibre's library location as well as the metadata.db database. I only backed up the database and I lost ~12 books in the process. I was able to recover 11 of them from non-Calibre sources, but I should've backed up first. Anyway, here's what I did:

1. Stop all running instances of Calibre, both on the WHS and whatever machine is going to be your new client.
2. On the WHS, move your calibre metadata.db database to a share that is writable by your client machine. I put mine in \\servername\software\Calibre\metadata.db, but you can put yours wherever you like. In my case, I keep my ebooks in \\servername\ebooks\calibre and I have my ebooks share set for duplication, while the software share is not set for duplication. I'm not sure if duplication plays nice with the database file and Calibre (see note below).
3. On both the WHS and the client machine, edit your environment variables to add CALIBRE_OVERRIDE_DATABASE_PATH pointing to wherever you put the database. I put this in the system-level environment variables for WHS, though I probably didn't need to (see below for why).
4. On the client machine, open Calibre, go into Preferences, and set the "Location of ebooks" value on the General tab to the share where your ebooks live. As I mentioned before, this was \\servername\ebooks\calibre for me. If you haven't done this on the WHS already, close Calibre on the client and update the server's settings as well. This is where something (I don't know what) went wrong and I lost a few books from my library (database entries were still there with metadata, but the files themselves were nowhere to be found). I think this might have been caused by having my ebook location set to a different value than what the database expected (?) when I started the client with the redirected database location, but I don't know why it only deleted some of my books. Anyway, if this happens to you, you can copy the files from your backup (you did backup above, right?) back into the correct location and everything should just work.

At this point, you now have Calibre setup on your client machine but with its database and files stored on the WHS. Next, we want to setup the calibre-server in order to provide access to Stanza, Trook, or just web browsing.

1. With Calibre closed on your client machine, open Calibre on the server.
2. Setup the port and other settings for the Content Server if needed and close Calibre.
3. Install the Windows Server 2003 Resource Kit and install it on your WHS.
4. Follow the instructions here to create a service for Calibre. I named my service simply "Calibre".
5. Before starting your service, you'll need to set the logon credentials. Local System won't work because Local System can't run a networked service. I tried Network Service instead but it didn't seem able to load the database (not sure if it's a permission issue or an environment variable issue, but this is why I set the environment variable globally above). I did get it to work by running it as administrator, but that's not a viable long-term option. Worst case, you may have to create a new user with limited permissions to run the service.
6. Start the service and then navigate to "http://servername:8888" (or whatever port you used) and see the magic happen.

Note about putting metadata.db on WHS shares: The DE migrator service can't move files that are locked by a process. Normally this isn't an issue because the service will keep trying over the next ~24 hours and at some point in there most processes will release whatever files the have locked. Calibre's GUI does not do this -- it locks the database as soon as it starts and holds onto it until it closes. If you're running the Calibre GUI 24/7 on your WHS like I was, within a day or so you'll start getting warnings on the WHS console about not being able to migrate files. In theory, since calibre-server only takes a read-only lock on the file the DE migrator should play nicely. However I'm not sure how nicely it will play with duplication so for now I've put my database on a non-duplicated share.

[solomon]

Originally Posted by toddos

Before starting your service, you'll need to set the logon credentials. Local System won't work because Local System can't run a networked service. I tried Network Service instead but it didn't seem able to load the database (not sure if it's a permission issue or an environment variable issue, but this is why I set the environment variable globally above). I did get it to work by running it as administrator, but that's not a viable long-term option. Worst case, you may have to create a new user with limited permissions to run the service.


<snip>

...for now I've put my database on a non-duplicated share.

Thanks a million! With this guide I was able to migrate my library in just a few minutes! I took a chance and left my metadata.db on the duplicated share to see what happens.... I'll move it if problems ensue.

Perhaps we can get Kovid to weigh in on why the service doesn't run properly as another user? I too don't like the idea of leaving it running as admin!

Richard

[kovidgoyal]

Sorry guys, I'm not a windows user, so I really have no idea why it will only run as admin

[solomon]

Originally Posted by kovidgoyal

Sorry guys, I'm not a windows user, so I really have no idea why it will only run as admin

Is there a log produced? I tried --develop and it seemed to log to STDERR which I wasn't able to capture as a service. (I could redirect it in a command shell easily enough, but that didn't seem to take as a service.)

I see the server process start and then exit, so I'm hoping if we could get a log we could figure out why it's unhappy.

Thanks again!
Richard

[toddos]

Well, I was able to get the error that calibre-server throws when it's run as local system or network service. I created a batch file that echos the value of CALBIRE_OVERRIDE_DATABASE_PATH and the output of calibre-server to a temp log file, and I got this:

d:\shares\software\calibre\metadata.db
Python function terminated unexpectedly
coercing to Unicode: need string or buffer, NoneType found (Error Code: 1)

I get the same error as local system or network service. I haven't jumped into the server code to understand where this is coming from yet.

[kovidgoyal]

No it logs only to stderr/stdout

[kovidgoyal]

It does log http access/errros but those logs wont help debug a startup problem. If you want to access them they'll be in the calibre config directory (accessible via preferences->advanced)

[toddos]

I'm a little bit closer, but this is turning into quite the mess. What I've done so far:

1. Moved configuration from the administrator's Application Data directory to another location using the CALIBRE_CONFIG_DIRECTORY environment variable
2. Gave Network Service rw access to the config location (don't know why it needs write access, but read didn't seem to be enough -- looks like it was related to the ExclusiveFile code that attempts to "lock" files so others can't read them).
3. Gave Network Service read access to the ebook repository (both local file access and share access, since I'm referring to the location by share name)

At this point I'm getting this error:

Python function terminated unexpectedly
unable to open database file
RemoteTraceback (most recent call last):
File "site-packages\calibre\library\sqlite.py", line 127, in run
OperationalError: unable to open database file (Error Code: 1)

Network Service has read/write access to the metadata.db database location. The GUI can still read the database so it's not corrupted.

I'm about ready to give up on this and just keep running the service as administrator. At least that worked

[solomon]

Originally Posted by toddos

I'm a little bit closer, but this is turning into quite the mess.

You're ahead of me again, I see that the global.py for NetworkService has None for library_path and language, and it overwrites those each time it launches.

7c7
< database_path = 'C:\\Documents and Settings\\NetworkService/library1.db'
---
> database_path = 'C:\\Documents and Settings\\Administrator/library1.db'
23c23
< library_path = None
---
> library_path = u'//Server/eBooks'
27c27
< language = None
---
> language = 'en'
55c55
< installation_uuid = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
---
> installation_uuid = 'yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy'


I don't understand the database_path bit at all. None of those files exist?

Confused,
Richard

[toddos]

The database path is a configuration item (not exposed in the UI for some reason) to move the database to a different location. I don't know if it's actually used anywhere, since it defaults to the user's profile directory + library1.db but the database always ends up in the root of the library path. I've been using the environment variable CALIBRE_OVERRIDE_DATABASE_PATH to move the database to a location other than the library path. I initially did that due to the DE migrator problem I mentioned earlier so that the GUI would not block WHS's drive rebalancing (moved the database to a non-share directory on c:, for example). It's potentially unnecessary with this approach if you don't mind your metadata.db living in the library root. Personally I like the idea of keeping the database separate from the files.

When I changed the config location above using the environment variable, I pre-populated that location using the administrator's configuration files.