What are the reasons calibre can't support multi-user libraries?

aurne · 03-15-2018, 07:32 PM

I'm planning on implementing better networked library support for calibre, essentially allowing multiple clients to connect to a centralized library.

As sqlite supports multiple client connections, albeit locks the whole database during writes, I'm wondering why multiple clients currently cause database corruptions and what code interfaces with the db (is it all in src/db/ or is there more).

Currently considering a client/server model where the server is the sole writer and processes db IO requests sent by clients.

Please let me know your thoughts, and if there is anything important I'm missing.

ilovejedd · 03-15-2018, 08:41 PM

I believe multiple users are already supported via the content server.

aurne · 03-15-2018, 08:59 PM

Quote:

Originally Posted by ilovejedd

I believe multiple users are already supported via the content server.

Unfortunately the content server is severely limited in terms of functionality such as editing book metadata (and thus database operations), compared to the desktop version.

BetterRed · 03-16-2018, 12:32 AM

@aurne - I believe one reason will be that the 'database' on which desktop calibre operates is not the on-disk metadata.db sqlite database as such. Instead it operates on an in-memory 'database' which is initially built from the on-disk sqlite metadata.db when calibre starts. See API documentation for the database interface

BR

aurne · 03-16-2018, 12:46 AM

Quote:

Originally Posted by BetterRed

@aurne - I believe one reason will be that the 'database' on which desktop calibre operates is not the on-disk metadata.db sqlite database as such. Instead it operates on an in-memory 'database' which is initially built from the on-disk sqlite metadata.db when calibre starts. See API documentation for the database interface

BR

Thanks! That's pretty informative, though it seems like a really non-standard way to handle databases. The wiki quotes "maximum performance" as justification. Do you, or anyone else have any numbers regarding the performance differential between in-memory and traditional sqlite?

Anyways, it seems like my original architecture of a single server performing db IO with multiple clients connected will be the best choice going forward.

BetterRed · 03-16-2018, 01:14 AM

@aurne - I'm sure Kovid will be along soon, and I've no doubt he'll give you a definitive response. It's an issue he has commented on at various times, sorry I don't have a link to hand. A search for client-server in the calibre forum might find something.

See also - Kovid's TODO list

BR

DaltonST · 03-16-2018, 08:46 AM

Quote:

Do you, or anyone else have any numbers regarding the performance differential between in-memory and traditional sqlite?

Aurne:

As a proxy answer, search for the reason that people use SSDs instead of Hard Drives.

DaltonST

DNSB · 03-16-2018, 12:47 PM

Quote:

Originally Posted by aurne

Thanks! That's pretty informative, though it seems like a really non-standard way to handle databases. The wiki quotes "maximum performance" as justification. Do you, or anyone else have any numbers regarding the performance differential between in-memory and traditional sqlite?

Not sure how relevant to sqlite this is but Microsoft's SQL 2014 and 2016 have the capability to use in-memory OLTP. Various tests showed a 30x increase in performance depending on what your task mix is. You just need lots of memory to hold the database -- in our case, the SQL servers have 1TB of memory with a flash SAN for disk storage.

kovidgoyal · 03-16-2018, 12:58 PM

The calibre database is perfectly multi-user, the calibre GUI is not. And the as of the next calibre release the content server will support editing metadata, which you can also already do using calibredb with a remote database.

sjfan · 03-16-2018, 03:12 PM

Quote:

Originally Posted by kovidgoyal

The calibre database is perfectly multi-user, the calibre GUI is not. And the as of the next calibre release the content server will support editing metadata, which you can also already do using calibredb with a remote database.

Just for clarity, are you saying it's safe to run calibredb against the same database while a GUI (or content server) is running on that database?

kovidgoyal · 03-16-2018, 03:19 PM

calibredb can talk to the calibre content server, wherever it is running, either standalone or inside a calibre GUI. In either case it is safe to do so, with the obvious caveat that if you do something like deleting a book while simultaneously editing metadata for the same book, you will get an error for one of the two operations, but the other one will succeed.

aurne · 03-16-2018, 07:38 PM

Quote:

Originally Posted by kovidgoyal

The calibre database is perfectly multi-user, the calibre GUI is not. And the as of the next calibre release the content server will support editing metadata, which you can also already do using calibredb with a remote database.

The change I am proposing is to allow the calibre GUI to interface with a remote database, possibly through calibredb, as the desktop GUI is quite powerful and intuitive.

Just to confirm, so I don't duplicate planned functionality, this is not the feature that will be present in the next release?

kovidgoyal · 03-17-2018, 02:45 AM

Quote:

Originally Posted by aurne

The change I am proposing is to allow the calibre GUI to interface with a remote database, possibly through calibredb, as the desktop GUI is quite powerful and intuitive.

Just to confirm, so I don't duplicate planned functionality, this is not the feature that will be present in the next release?

No, it is not.

eschwartz · 04-22-2018, 01:18 PM

Quote:

Originally Posted by aurne

The change I am proposing is to allow the calibre GUI to interface with a remote database, possibly through calibredb, as the desktop GUI is quite powerful and intuitive.

Just to confirm, so I don't duplicate planned functionality, this is not the feature that will be present in the next release?

It is in fact what is referenced in the TODO which BetterRed linked, as:

Quote:

(2) is preferable in terms of features/functionality, but is much harder to implement in a performant and semantically correct manner.

As far as I am aware, Kovid has no current plans implementing either one, hence why it is dropped unceremoniously into the TODO until he can figure out what to do with them.

It's a bit of a challenging issue as indicated... if you can implement this I'm sure lots of people will be quite interested.

03-15-2018, 07:32 PM	#1
aurne Junior Member Posts: 4 Karma: 10 Join Date: Mar 2018 Device: kindle	What are the reasons calibre can't support multi-user libraries? I'm planning on implementing better networked library support for calibre, essentially allowing multiple clients to connect to a centralized library. As sqlite supports multiple client connections, albeit locks the whole database during writes, I'm wondering why multiple clients currently cause database corruptions and what code interfaces with the db (is it all in src/db/ or is there more). Currently considering a client/server model where the server is the sole writer and processes db IO requests sent by clients. Please let me know your thoughts, and if there is anything important I'm missing.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
HarperCollins to Offer Multi-User E-Book Access to Libraries	tubemonkey	News	11	07-07-2017 10:32 PM
Multi-user calibre library	malamona	Development	3	07-11-2013 12:48 PM
Google Play store & multi-user support	avantman42	Kindle Fire	11	04-11-2013 09:17 PM
Multi language support via Calibre	Jim77	Calibre	4	02-12-2010 03:46 PM

03-15-2018, 08:41 PM	#2
ilovejedd hopeless n00b Posts: 5,110 Karma: 19597086 Join Date: Jan 2009 Location: in the middle of nowhere Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9	I believe multiple users are already supported via the content server.

03-16-2018, 12:32 AM	#4
BetterRed null operator (he/him) Posts: 21,004 Karma: 27620706 Join Date: Mar 2012 Location: Sydney Australia Device: none	@aurne - I believe one reason will be that the 'database' on which desktop calibre operates is not the on-disk metadata.db sqlite database as such. Instead it operates on an in-memory 'database' which is initially built from the on-disk sqlite metadata.db when calibre starts. See API documentation for the database interface BR

03-16-2018, 01:14 AM	#6
BetterRed null operator (he/him) Posts: 21,004 Karma: 27620706 Join Date: Mar 2012 Location: Sydney Australia Device: none	@aurne - I'm sure Kovid will be along soon, and I've no doubt he'll give you a definitive response. It's an issue he has commented on at various times, sorry I don't have a link to hand. A search for client-server in the calibre forum might find something. See also - Kovid's TODO list BR

03-16-2018, 12:58 PM	#9
kovidgoyal creator of calibre Posts: 44,556 Karma: 24495948 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The calibre database is perfectly multi-user, the calibre GUI is not. And the as of the next calibre release the content server will support editing metadata, which you can also already do using calibredb with a remote database.

03-16-2018, 03:19 PM	#11
kovidgoyal creator of calibre Posts: 44,556 Karma: 24495948 Join Date: Oct 2006 Location: Mumbai, India Device: Various	calibredb can talk to the calibre content server, wherever it is running, either standalone or inside a calibre GUI. In either case it is safe to do so, with the obvious caveat that if you do something like deleting a book while simultaneously editing metadata for the same book, you will get an error for one of the two operations, but the other one will succeed.

Advert

Advert