Calibre Companion: upcoming calibre 1.13 release forces full metadata send

[chaley]

I have changed calibre (and CC) to store metadata in calibre on a per-device basis. This change fixes several small problems. Examples:
- refreshing metadata on device change if the thumbnail size is different on the devices
- refreshing metadata when metadata is changed in calibre then sent to one device, then later another.
- confusion when identical books have different file locations on different devices

Other changes fix a problem that arises if the identical book is somehow added to CC more than once with different file names.

One consequence of the changes: the first CC connect after upgrading will do a complete metadata send to calibre. Keep that in mind when you choose when to upgrade.

If you upgrade calibre to 1.13 but have not upgraded CC to 3.2.5, the next CC upgrade will also force a full metadata send. You can avoid this by ensuring that CC is fully upgraded before connecting to the upgrading calibre.

[PatNY]

Hi Charles,

Thanks for the warning as I tend to get impatient and think something is broken if a process is taking too long or the screen is stuck on the same message.

I have a question about syncing with multiple Calibre libraries. Currently I sync two Nook devices with 2-3 different Calibre libraries using both Wi-fi sync, and also occasionally just import via the content server to both devices. I have my main books in library 1 and just magazines and periodicals in library 2. Library 3 has additional magazines and periodicals. The mix of books and publications on my two devices are different too. Do I ever have to worry about things (like metadata) getting mixed up in CC? If I move some items in library 3 that were previously synced to one of my devices into library 2 and do a subsequent sync, will CC ever get confused? Just wondering how far to push things and trying to avoid potential problems.

--Pat

[chaley]

Originally Posted by PatNY

Hi Charles,

Thanks for the warning as I tend to get impatient and think something is broken if a process is taking too long or the screen is stuck on the same message.

You are welcome. Avoiding confusion and concern was my goal.

I have a question about syncing with multiple Calibre libraries. Currently I sync two Nook devices with 2-3 different Calibre libraries using both Wi-fi sync, and also occasionally just import via the content server to both devices. I have my main books in library 1 and just magazines and periodicals in library 2. Library 3 has additional magazines and periodicals. The mix of books and publications on my two devices are different too. Do I ever have to worry about things (like metadata) getting mixed up in CC? If I move some items in library 3 that were previously synced to one of my devices into library 2 and do a subsequent sync, will CC ever get confused? Just wondering how far to push things and trying to avoid potential problems.

This question is very hard to answer succinctly, so you are going to get a core dump.

First, I am assuming that all these libraries are managed by the same calibre installation. If they are not then the behavior of the metadata cache in calibre will depend on which installation is used. (Hmmm... I need to think about this. I wonder if I should have a cache per device per library?)

Calibre's device system matches books (a "book" is anything stored in calibre) on the device to those in calibre first using the book's calibre unique identifier (UUID). If there is no match, calibre falls back to title/author matching.

A Book's UUID depends on the library the book is in. If you copy the book to another library, it will get a new UUID in that other library. The books are no longer the same. They can have different custom columns, different actual metadata, and different modification times.

Putting all this together, we can predict the following behavior:

• If a book in CC is not in the current library (different UUID and non-matching title/author), then that book's metadata will not be updated. It will appear in the device view but won't have any On Device/In Library check marks.
• If a book in CC is matched to the current library by UUID, the book's metadata will be updated and will have check marks. This is the behavior we almost certainly want.
• If a book in CC is matched to the current library by title/author, then the metadata in CC will be updated to equal that of the matched book. Thus, if the same book is in two libraries, the metadata in CC will probably be from the last library you connected to. The book will have check marks.
• Plugboards can get in the way because they often change the title and author, making books not match if their UUIDs change. Because of this, moving a book to another library may "disconnect" that book from CC's point of view, resulting in a first-case (no match) situation.
• If a book on the device is not matched and is sent again, then the book may be on the device twice. The "may be" arises because the file names of the books on the device may be different (send to device template using different metadata), preventing overwriting the first book by the second.

The plugboard case is fixed by manually matching the books in the new library. You do this in the device view, telling calibre that this book (format) on the device is really that book in the library.

The third case, title/author matching, is potentially problematic, such as when the libraries have different books that for some reason have the same title/author. For example, if you keep a library for books in English and another for books in French, and if the titles & authors are untranslated, then the metadata for the English version could get updated with the French commentary. If the books in the two libraries are really the same, then the side effects of matching will be limited to differences in metadata and custom column contents.

The last case, multiple copies of the book, is probably only an annoyance, but could be confusing if you don't notice that there are two copies. Metadata for one of the copies would be updated when connecting to its library, while metadata for the other would be updated when connecting to its (separate) library. It would be quite difficult for you to know which copy you are looking at.

I have worried about device support for multiple libraries ever since multiple libraries were added to calibre. As discussed above, the many-library to one-device mapping can produce various hard-to-predict anomalies. For file-system-based devices, there is no choice but to live with this. For CC, I could imagine building separate library support into the app, but I am almost certain that this cure would worse than the disease.

Will you have problems? I don't know. Probably not if you guarantee that a book is in exactly one library and that books in different libraries will never accidentally have the same title/author. If you move books then you might need to do some manual book matching in the destination library.

[PatNY]

Hi Charles, thanks for the detailed answer. Based on it, I think I can go on syncing multiple devices with multiple libraries without issues. My setup: All Calibre libraries are under a single installation, I do not use plugboards, and all books across the multiple libraries each have a unique title, even if they are the same literary work.

I think the logic in CC is well thought out -- and safe, at least for the way I use it -- and I have peace of mind knowing I will not end up with a jumbled mess one day.

BTW, initial syncs with 1.13 were of normal length for my setup and didn't take a long time like it has in the past.

--Pat