Manage Collections

[timlegge]

I am trying to figure out how to add collections support to the Kobo.

What is it that the manage collections action from the device popup menu does?

I assume that it would add the currently selected collection to the currently selected book. Is that correct?

I will probably have other questions as I work on this...

Tim

[chaley]

Originally Posted by timlegge

I am trying to figure out how to add collections support to the Kobo.

What is it that the manage collections action from the device popup menu does?

I assume that it would add the currently selected collection to the currently selected book. Is that correct?

I will probably have other questions as I work on this...

Tim

I assume you are asking about something calibre does, and not what the kobo does?

From the point of view of a device driver, collection management is currently arranged in the class CollectionsBookList. There is one interface method, rebuild_collections, and one utility method, get_collections. Both of these interact with the Metadata Management setting in preferences -> add/save -> send to device.

When a device is plugged in: if a device supports collections, then the 'books' interface method *must* ensure that the book.device_collections contains a list of the collections of which the book is a member. If it does not, and if metadata management is set to manual, then the collections will be lost. In addition, the GUI will not be able to display the information.

If Metadata management is manual, then the user can add and remove collections on the device view. After editing, calibre modifies book.device_collections and calls rebuild_collections with that book (or books). Rebuild_collections must do whatever is necessary to make the device's collections match the contents of the 'device_collections' attribute. It is not necessary (nor desirable) to call booklist.get_collections

Devices that support collections should call booklist.get_collections whenever metadata is changed, i.e. when sync_booklists is called. Get_collections returns a dictionary {collection:[list of books], c:[bl], ...}. The list of books is in the order that they should appear on the device. The device driver's job is to ensure that collections on the device match what is in the dict, deleting and adding as needed. Get_collections uses the Metadata management setting, the new_book flag, and the book's device_collections attribute to build the dict.

If metadata management is set to either automatic or on_send, then collections are built exclusively from the calibre attributes specified in the device customization. Custom field attributes are being added real-soon-now. The user cannot add or remove collections on the device view.

The Sony drivers are a good but complicated example of how collection management works. It is very possible that some sony-only stuff has crept into the general code. Don't hesitate to point that out. It would be best to try to avoid creating another collection management structure.

[timlegge]

Originally Posted by chaley

I assume you are asking about something calibre does, and not what the kobo does?

Yes

[snip]

Originally Posted by chaley

When a device is plugged in: if a device supports collections, then the 'books' interface method *must* ensure that the book.device_collections contains a list of the collections of which the book is a member. If it does not, and if metadata management is set to manual, then the collections will be lost. In addition, the GUI will not be able to display the information.

I am okay there, the Kobo has one collection the "I'm Reading" list (although it may be possible to work with a few other things) that is held as ReadStatus in the sqlite database. When I process the books, I update the device_collections. Unfortunately, at the moment it is not getting stored in the metadata.

Originally Posted by chaley

If Metadata management is manual, then the user can add and remove collections on the device view. After editing, calibre modifies book.device_collections and calls rebuild_collections with that book (or books). Rebuild_collections must do whatever is necessary to make the device's collections match the contents of the 'device_collections' attribute. It is not necessary (nor desirable) to call booklist.get_collections

How do you add or remove collections on the device view? The only menu item I have is manage collections and it seems to only allow me to remove collections from all books or rename a current collection. Did I miss setting a value to enable a menu item?

Originally Posted by chaley

If metadata management is set to either automatic or on_send, then collections are built exclusively from the calibre attributes specified in the device customization. Custom field attributes are being added real-soon-now. The user cannot add or remove collections on the device view.

So, if I have a tag applied to the book for "I'm Reading" (and its configured to look at the tags) any books with the I'm Reading tag will have its metadata set to "I'm Reading" I see the menu disappear for Manage Collections when I select the On Send or Auto so I see how that part works.

Originally Posted by chaley

The Sony drivers are a good but complicated example of how collection management works. It is very possible that some sony-only stuff has crept into the general code. Don't hesitate to point that out. It would be best to try to avoid creating another collection management structure.

Yes, I have been studying the prs505 driver. Basically, I plan to implement with rebuild_collections and the current collection management...

Tim

[chaley]

Originally Posted by timlegge

Unfortunately, at the moment it is not getting stored in the metadata.

Some initial comments:

1) Be sure that you set the booklist_class variable to CollectionsBookList. If you don't, then calibre thinks that the device cannot support collections.
2) Don't hack too much on the current driver. Kovid and I are in process of changing how book metadata is stored on devices, which includes how json is handled. The kobo driver will need changes (I did some so it will compile). My guess is that you will be able to get hold of the new stuff within a week or so.

Is ReadStatus a boolean? If so, then you can store it in device_collections as a single value. Something like
if ReadStatus == True: device_collections = ['Reading']
else: device_collections = []

The other way would work something like
if 'Reading' in device_collections: ReadStatus = True
else: ReadStatus = False

Of course, you can map a tag to ReadStatus, but you will still need to set device_collections.

How do you add or remove collections on the device view?

You double-click on a book in the collections column. However, that won't work unless you have done #1 above. In fact, you shouldn't see the menu if #1 isn't done.

So, if I have a tag applied to the book for "I'm Reading" (and its configured to look at the tags) any books with the I'm Reading tag will have its metadata set to "I'm Reading" I see the menu disappear for Manage Collections when I select the On Send or Auto so I see how that part works.

I am not sure what you are saying here. The book will contain an attribute 'tags' that is a list of tags attached to the book. It is up to the driver to do something with that, and to ensure that device_collections is set properly.

[timlegge]

Originally Posted by chaley

Some initial comments:

1) Be sure that you set the booklist_class variable to CollectionsBookList. If you don't, then calibre thinks that the device cannot support collections.

Done, I have been seeing the Manage Collections item on the popup menu and the Collections column however, double-clicking on the book in the Collections column does not do anything.

Originally Posted by chaley

2) Don't hack too much on the current driver. Kovid and I are in process of changing how book metadata is stored on devices, which includes how json is handled. The kobo driver will need changes (I did some so it will compile). My guess is that you will be able to get hold of the new stuff within a week or so.

My guess is that I will not get much time to hack this week, thanks for the heads up.

Originally Posted by chaley

Is ReadStatus a boolean? If so, then you can store it in device_collections as a single value. Something like
if ReadStatus == True: device_collections = ['Reading']
else: device_collections = []

The other way would work something like
if 'Reading' in device_collections: ReadStatus = True
else: ReadStatus = False

playlist_map = {}

if readstatus == 1:
if lpath not in playlist_map:
playlist_map[lpath] = []
playlist_map[lpath].append("I\'m Reading")

and later:
book.device_collections = playlist_map.get(book.lpath, [])

That does display "I'm Reading" in the correct books in the gui.

Originally Posted by chaley

Of course, you can map a tag to ReadStatus, but you will still need to set device_collections.

I am not sure how to map tags.

Originally Posted by chaley

You double-click on a book in the collections column. However, that won't work unless you have done #1 above. In fact, you shouldn't see the menu if #1 isn't done.

Done and not working :-(

Originally Posted by chaley

I am not sure what you are saying here. The book will contain an attribute 'tags' that is a list of tags attached to the book. It is up to the driver to do something with that, and to ensure that device_collections is set properly.

I am still trying to understand how the tags interact. I am assuming that if I tag a book in the Library with an I'm Reading tag that the collections functionality can use that to create a Device Collection (if the driver is told to do something with that tag). I just need to parse the tags and ensure that the metadata file holds the device collections and that the database on the device is updated to set the reading status to 0 or 1 depending on the tags for the books. I may be missing the point totally...

Tim

[chaley]

Originally Posted by timlegge

Done, I have been seeing the Manage Collections item on the popup menu and the Collections column however, double-clicking on the book in the Collections column does not do anything.

Ahhh... you also need to add
CAN_SET_METADATA = True
in your driver. Try putting it after line 29
FORMATS = ['epub', 'pdf']

playlist_map = {}

if readstatus == 1:
if lpath not in playlist_map:
playlist_map[lpath] = []
playlist_map[lpath].append("I\'m Reading")

and later:
book.device_collections = playlist_map.get(book.lpath, [])

That does display "I'm Reading" in the correct books in the gui.

Good. At least that much is working.

I am still trying to understand how the tags interact. I am assuming that if I tag a book in the Library with an I'm Reading tag that the collections functionality can use that to create a Device Collection (if the driver is told to do something with that tag). I just need to parse the tags and ensure that the metadata file holds the device collections and that the database on the device is updated to set the reading status to 0 or 1 depending on the tags for the books. I may be missing the point totally...

You have it right. That said, you need to decide what functionality you are trying to support.

You have already shown that you can create a collection from the Kobo's I'm_Reading flag, and that the user can see this collection on the device view. Once you add the CAN_SET_METADATA, the user will be able to change that field. The question is 'what is supposed to happen'? For example, what if the user adds 'Foobar' to the collection list? What is the Kobo supposed to do with that? What if the user adds 'I'm Reading'? Is the Kobo supposed to set the bit in the database? If the user sets a tag 'Reading' (or some such), is the Kobo driver supposed to turn the bit on? Is it possible to use the Kobo to turn the bit off? If so, what is calibre supposed to do?

My guess is that as a first iteration, you should report the I'm Reading status back to the user in a collection. Second iteration: allow the user to edit a book's collections, either adding or removing I'm Reading, than having the Kobo driver either set or clear the bit. We might want to add another interface method so that the GUI can enquire of the driver whether the collection list is valid for the device, to handle the Foobar case. Third iteration: associate a tag with I'm Reading. If the tag is present, then set the Kobo's bit. If the tag is absent, then clear the Kobo's bit. You may want to require that Manage Metadata be set to Automatic, so that editing the collections on the device view is prohibited. Otherwise there will be unpleasant interactions between the processing of the tag and the user's manual changes. Just random thoughts...

[chaley]

By the way, if you set CAN_SET_METADATA = True, then the user will be able to change author and title on the device view. Can the Kobo cope with that? Are author and title stored in the DB, independently from the book file?

[timlegge]

Originally Posted by chaley

Ahhh... you also need to add
CAN_SET_METADATA = True
in your driver. Try putting it after line 29
FORMATS = ['epub', 'pdf']

Good. At least that much is working.

Thank you !!!! That would have been tough to find...

Originally Posted by chaley

You have it right. That said, you need to decide what functionality you are trying to support.

You have already shown that you can create a collection from the Kobo's I'm_Reading flag, and that the user can see this collection on the device view. Once you add the CAN_SET_METADATA, the user will be able to change that field. The question is 'what is supposed to happen'? For example, what if the user adds 'Foobar' to the collection list? What is the Kobo supposed to do with that? What if the user adds 'I'm Reading'? Is the Kobo supposed to set the bit in the database? If the user sets a tag 'Reading' (or some such), is the Kobo driver supposed to turn the bit on? Is it possible to use the Kobo to turn the bit off? If so, what is calibre supposed to do?

Good questions. I did note and saw your post that the other fields become changeable too. Initially I ,am most interested in updating the sqlite database on the device if the device collection is set to I'm reading. I would be fine with saving anything else that the user cared to place in the field to the metadata.calibre file without doing anything else with it.

The kobo is locked while connected to the computer (at present). The kobo clears the I'm reading flag when a book is finished. My plan is that calibre will read the flag from the database when it is connected so the collection will always be whatever the kobo says at initial connection. Then calibre will set the flag depending on the users settings I guess. On auto I assume it would rebuild the device collections based on the tagged files. I will need to think on it because I don't really want to overwrite someones leading list if they don't want it overwritten.

Originally Posted by chaley

My guess is that as a first iteration, you should report the I'm Reading status back to the user in a collection. Second iteration: allow the user to edit a book's collections, either adding or removing I'm Reading, than having the Kobo driver either set or clear the bit. We might want to add another interface method so that the GUI can enquire of the driver whether the collection list is valid for the device, to handle the Foobar case. Third iteration: associate a tag with I'm Reading. If the tag is present, then set the Kobo's bit. If the tag is absent, then clear the Kobo's bit. You may want to require that Manage Metadata be set to Automatic, so that editing the collections on the device view is prohibited. Otherwise there will be unpleasant interactions between the processing of the tag and the user's manual changes. Just random thoughts...

Thanks. That roughly follows what I am thinking. For the kobo, calibre's killer feature will probably be reseting the I'm reading list. The device sets the flag when ever any book is open and allows no way to manually stop reading a book...

Tim

[timlegge]

Originally Posted by chaley

By the way, if you set CAN_SET_METADATA = True, then the user will be able to change author and title on the device view. Can the Kobo cope with that? Are author and title stored in the DB, independently from the book file?

Those fields are all stored in the database separate from the pubs. There is no reason it could not be supported if I have enough time/Interest...

[timlegge]

I have managed to add simple support for device collections which seems to work okay for manual editing. It still needs some work and I will probably wait till you merge the new metadata/json code and rework it then.

I did notice a couple of things:

It seems that device_collections needs to be added to the JSON_ATTRIBUTES in usbms/books.py or the collection is not written to metadata.calibre

There maybe a bug in the current driver that will cause a failure if a new Kobo is used with Calibre (I need to retest to be sure).

rebuild_collections receives the entire booklist when a device collection is changed. In the case of the Kobo, that leads me to reset and recreate the the I'm Reading list everytime one collection is changed. It does not take long and I may be able to query the device for the collections and only reset the ones that are no longer on the list but that may well be slower.

I have not looked at the on send setting and automatic settings. Do they still call rebuild_collections?

Tim

[timlegge]

Originally Posted by timlegge

I have not looked at the on send setting and automatic settings. Do they still call rebuild_collections?

Looks like "Only on send" makes use of sync_booklists

[timlegge]

I have a very rough version of the driver that can both manually manage the I'm Reading list and do it automatically on connecting the device. There is still a fair amount of work and some functionality (like manual management) may be dropped.

I do like plugging it into Calibre and getting the Reading List overwritten by any books tagged with "I'm Reading". However, getting a book that you are currently reading on the device removed from the reading list (because its not in the Library or tagged) may be less than ideal. What is Calibre's preferred method? Should I overwrite or merge the reading list.

Tim

[kovidgoyal]

the metadaa management options are for that. Manual means dont overwrite device collections. Automatic means overwrite device collections.

[chaley]

You have several questions in this and later posts. I will try to deal with them all here.

Originally Posted by timlegge

I will probably wait till you merge the new metadata/json code and rework it then.

That is probably wise.

It seems that device_collections needs to be added to the JSON_ATTRIBUTES in usbms/books.py or the collection is not written to metadata.calibre

Device_collections is to be recomputed on each connect (in the driver.books method), which is why it is not written to the cache. Reason: the values can (and do) change, so the cached value could not be trusted.

rebuild_collections receives the entire booklist when a device collection is changed. In the case of the Kobo, that leads me to reset and recreate the the I'm Reading list everytime one collection is changed. It does not take long and I may be able to query the device for the collections and only reset the ones that are no longer on the list but that may well be slower.

Again, you are right. My thought was that there may be interactions between collections (cross-links or something), and that rebuild_collections should have the entire list to work with.

I have not looked at the on send setting and automatic settings. Do they still call rebuild_collections?

No. They affect when sync_booklists is called, and what get_collections does.

Manual: sync_booklists is called only when a book is added or removed. Get_collections will compute collections only for books with 'new_book == True'.

On_send: sync_booklists is called only when a book is added or removed. Get_collections will recompute all the collections for all books on the device.

Automatic: sync_booklists is called on every connect. Get_collections will recompute all the collections for all books on the device.

Looks like "Only on send" makes use of sync_booklists

Sync_booklists is called whenever a book is added or removed. It is to do everything necessary to ensure that the metadata on the device is consistent with the metadata in the booklists. Collection processing is guided as described above.

On the Sony, title & author & author_sort are updated for each book every time sync_booklists is called. This is done because the Sony can decide to reset the author value to what is in the book, undoing the user's request to use author_sort for the author. It can also undo setting the titleSorter field.

For the Kobo, you could choose to use the _new_book flag and the setting of Metadata Management to decide whether or not to update authors etc.

Rebuild_collections is called only when collections are edited on the device view. It should not touch any other metadata. This differs from sync_booklists, which has a free hand to change any metadata as needed.

I do like plugging it into Calibre and getting the Reading List overwritten by any books tagged with "I'm Reading". However, getting a book that you are currently reading on the device removed from the reading list (because its not in the Library or tagged) may be less than ideal. What is Calibre's preferred method? Should I overwrite or merge the reading list.

As Kovid said, use the value of Metadata Management to decide.

If it is manual, then you do a merge. Never delete a value. Get_collections respects this rule.

If it is on_send or automatic, then calibre's metadata wins. No merging is done. The user must ensure that the tag (or what-have-you) is correct.

One of the differences between on_send and automatic can be seen here. Since on_send does not change the metadata on connect, the user can see the current values in 'Device Collections'. That information can be used to by the user to do a manual merge, setting/clearing the tag as desired. When a book is sent, the tag will win. On 'automatic', the device is asked to change the values on connect, so the existing values will be overwritten. Calibre's data wins.