Kobo Device Driver Update

[kovidgoyal]

Just google usb sniffer or usb monitor, it should give you plenty of hits. I remember doing this about 6 years ago when I was reverse enginnering the USB protocol the sony prs 500 used to write the first calibre driver, but I cannot recall the tools I used.

[PeterT]

Originally Posted by ichrispa

Agreed. But even when nickel reopens the database it does not update the information displayed, suggesting as Davidfor mentioned that parts of the database are actually cached beyond the sqlite database system. As I mentioned I have to restart my KT to see new shelves, even if they are in the DB after a disconnect. Same goes for series information after they are inserted into the db: they are there, but only visible after a reboot.

In any case this is not a problem of either Calibre or the KT driver. It is up to Kobo to fix close the database properly and reload its information after a disconnect.



Apart from monitoring the USB bus itself (as in "capture with oscilloscope"), I have no idea how to intercept USB data from a virtual machine. Linux won't let me tap the system bus interface. And I definetely don't know how to do this in a native windows environment. Since I haven't managed to get Kobo Desktop to run in WINE, that's out of the question too.

Can somebody advise me on how to monitor what an application sends over USB?

My feelings ate that this might be related to how you have hacked the Kobo to get command line access... How come those of us who have not taken that step do not have this issue...

[Terisa de morgan]

I have a question. Are you using Linux as SO? I think so after your last message (I was doubtful before). So, is the problem really in Kobo FW or its interaction with Linux? I've never had this problem, and my Mini and my Glo interact a lot with calibre.

[davidfor]

@ichrispa: That is very interesting analysis, but like PeterT and Terisa, I have to disagree with it.

Firstly, I have to take something back. Or maybe modify it. I said that the device is caching the database and not necessarily rereading it after a connection to USB. I said this because I had a circumstance under which I modified the database and it wasn't shown in the library list. But, I found out why. I had misunderstood a column and hadn't changed it appropriately. Now that I fixed that, the changes are always showing up.

What I see is that if I connect my Glo or Touch to my Windows machine and fiddle with the database on the device, the changes are shown as soon as I disconnect. The "fiddling" could be anything. Adding rows to different tables or deleting them. The most common change is modifying the values in the content table. If I change the series info, the description, the last read date, the reading status, the last chapter read, the ISBN, the published date, the author, the title, the reading percentage or the file size, they are shown when I look at the device without restarting it. The same goes for adding a shelf or putting a book on a shelf. Or changing the stored font size for reading a book. Some of these changes are through the driver, some manually using a SQLite browser. A lot are via another different plugin I'm developing whose whole reason is to make these changes.

On a more sever thing to do, yesterday I removed all books from my Glo while it was connected, and replaced the database with one from another user. The I disconnect. The device opened and worked correctly. I couldn't open books but I could look at the details. Then I connect and added a book the user was having trouble with and again, no problems. Then I removed that database and put my 200 books and database back on. When I disconnected, it all worked perfectly.

I do see problems. And they are related to corrupt databases. But as far as I can tell, it is always my fault. I have calibre open, plus the database open and hit the eject button I have in calibre (a great plugin) and just pull the cable forgetting the database is still open and the device wasn't actually ejected from Windows. These cases are why I think the database might be cached. But I think what is happening is that when the device is disconnected, it tries to read the database. If it can't for some reason, it continues using the in-memory data. I don't know if it is using a memory copy of the database, or the built data structures. In these cases, any changes made while using the device (book reading status changed) are lost when the device is powered down and restarted. Or they don't appear in the database when I connect to the PC and look.

I don't doubt you are seeing what you report. But, it disagrees with my experience and the experiences reported here. But, I also don't know how many of them are using Linux. I know that people are using Windows and Macs with no problems, but I am not as a sure about Linux. I have definitely seen others using Linux with their Kobo device, but I am not as how many also use calibre. I might have go with it over the weekend if I have time. I don't currently have Linux box, but there is an idle laptop that could be changed. I'll see what happens.

And for the record, my Glo was bought in December last year. The Touch a year before. Both have had a factory reset in the last month followed by an upgrade to the firmware 2.4.0. And both had a logout last weekend which put a new database onto them and re-added all the books.

[PeterT]

Or just install VMware and then a Linux system under it.

I did try monitoring the calibre SysInternals ProcMon and validated that there we're no files open on the Touch before the eject was done.

[ichrispa]

Originally Posted by PeterT

My feelings ate that this might be related to how you have hacked the Kobo to get command line access... How come those of us who have not taken that step do not have this issue...

I frankly do not see the relation between the problem and the hack. Besides, the tests I described where done on an unhacked version (Hard Reset + FW 2.4.0 update).

Originally Posted by Terisa de morgan

Are you using Linux as SO? I think so after your last message (I was doubtful before). So, is the problem really in Kobo FW or its interaction with Linux? I've never had this problem, and my Mini and my Glo interact a lot with calibre.

Yes I am using Linux (for about 16 years now). Giving Linux the blame for this is a bit far fetched in my opinion, as everything performs as expected.

I have established that it is nickel (GUI on the Kobo) that does not behave properly. But I will try using Windows with calibre just to check the effect in the coming week.

Originally Posted by davidfor

What I see is that if I connect my Glo or Touch to my Windows machine and fiddle with the database on the device, the changes are shown as soon as I disconnect.

Thank you again for the work you are putting into this Davidfor. What you describe is definetely not what I am seeing. Even if I perform manual changes on the sqlite database and don't use calibre at all, the changes do not show up until the device is rebooted - and manual changes corrupt my database as well at times.

Your test is interesting in that I would love my KT to behave that way too... The most reliable way to kill my database is to connect, upload multiple ebooks and disconnect. After that, the database is gone.

Originally Posted by davidfor

I don't doubt you are seeing what you report. But, it disagrees with my experience and the experiences reported here. But, I also don't know how many of them are using Linux.

I am glad I am the only one with this problem. Again, I do not see how Linux would affect this. The FS is mounted and unmounted properly and the VFAT module certainly stable. Even the output from the ahci driver does not suggest anything funny going on.

Besides, as I said, I confirmed that nickel keeps its database open. If it is of interest, I will of course post how this was confirmed.

Originally Posted by PeterT

I did try monitoring the calibre SysInternals ProcMon and validated that there we're no files open on the Touch before the eject was done

Thanks for the trouble PeterT, but I fear you misunderstood my claim. On the Linux side the FS is unmounted properly, without any open files. What I found is that the (or apparently my) Kobo firmware does not release the database when the usb connection is established.


Again: This problem appears to be an isolated case of my KT behaving different from all of its kin. I have found a workaround. I am also writing a little python launcher to automate the steps described in the workaround before starting calibre.

If noone else is able to reproduce the problem, I would simply close the case and consider my KT very special

[davidfor]

Calibre 0.9.28 contains updates to the Kobo driver to add support for firmware 2.5.1 and the new Kobo Aura HD.

The only changes in function are:

- When a Kobo device is attached, the device name is displayed at the bottom of calibre. Previously this just had "Kobo Touch".

- Firmware 2.5.1 saves cover images on the SD card for books on the SD card. The driver will now send covers to the SD card when sending books to the SD card if the appropriate cover options are used.

[jackie_w]

@davidfor,

I've been using v0.9.28 with my Kobo Glo today and am experiencing a few teething problems. Here is a brief summary:

1. I only just updated to fw2.5.1 from v2.1.5 so this is the first time I've had the 'Set Series info' box checked. For books which don't have Series data, many of them have their Kobo Series field populated with the first char of the book Title after the follow-up connect/disconnect cycle.
   
   I've tried removing these books (via the Kobo) and resending, but the results are the same.
   
   
2. When reconnecting after send-to-device two books do not register a tick in the 'On device' column even though they are still on the Kobo. These particular books are both part of a series (2 diff authors, 2 different series). Because of this the Series data never gets transferred. For both author/series, I also sent other books in the same series which worked OK. Again, I went through the remove-resend loop but the results were always the same.
   
3. When using View details - Details option from the Kobo library list, I thought the Publisher field used to show the date corresponding to calibre's 'pubdate' field. It now always appears to be blank.

I realise that there probably isn't enough info here but before I write an essay I thought I'd just outline the situation in case any are known issues. I can provide more details if you can let me know what extra info/files you need to investigate further when you get some free time.

Thanks.

[GeoffR]

Originally Posted by jackie_w

[*] When using View details - Details option from the Kobo library list, I thought the Publisher field used to show the date corresponding to calibre's 'pubdate' field. It now always appears to be blank.

AFAIK this was a change in the firmware not in Calibre, since 2.1.5 the firmware no longer reads the published date from sideloaded epub metadata. You can enter the information into the Kobo database (as well as publisher and ISBN) and it will be displayed, but it is just not loaded into the database automatically by the firmware anymore.

[jackie_w]

Thanks for the quick response, Geoff. I assumed that calibre metadata was being written into the Kobo database as part of the 'Set series info' option. Is it not possible to transfer over other fields at the same time? Or am I completely misunderstanding the process?

[GeoffR]

Originally Posted by jackie_w

Thanks for the quick response, Geoff. I assumed that calibre metadata was being written into the Kobo database as part of the 'Set series info' option. Is it not possible to transfer over other fields at the same time? Or am I completely misunderstanding the process?

It is possible, the current driver just doesn't do it.

[davidfor]

Originally Posted by jackie_w

@davidfor,

I've been using v0.9.28 with my Kobo Glo today and am experiencing a few teething problems. Here is a brief summary:
[*]I only just updated to fw2.5.1 from v2.1.5 so this is the first time I've had the 'Set Series info' box checked. For books which don't have Series data, many of them have their Kobo Series field populated with the first char of the book Title after the follow-up connect/disconnect cycle.

I've tried removing these books (via the Kobo) and resending, but the results are the same.

I think this just proves you should have stayed with 2.1.5

What is the save template you are using? There is something that happens with a save template that includes series info and books that don't have series. But, I haven't seen it set the series on the device.

Could you run a debug session? Send a book without series info, disconnect, set the debug title in the driver options to the book name, restart in debug mode and reconnect. Then send me the log.

[*]When reconnecting after send-to-device two books do not register a tick in the 'On device' column even though they are still on the Kobo. These particular books are both part of a series (2 diff authors, 2 different series). Because of this the Series data never gets transferred. For both author/series, I also sent other books in the same series which worked OK. Again, I went through the remove-resend loop but the results were always the same.

Do these have multiple authors?

[*] When using View details - Details option from the Kobo library list, I thought the Publisher field used to show the date corresponding to calibre's 'pubdate' field. It now always appears to be blank.

The firmware stopped reading the date from epubs with 2.3.1. I think they stopped reading the publisher a little earlier. As you said in your other post the driver could do this when it is setting the series info. I have just been putting it off.

What I can do is to change the "Set series info" option to "Set other metadata". Then have it set the series info, publisher, published date and ISBN. I already have code to do most of it. The only concern is performance. The first connect after sending a lot of books will take more time, but after that, it should be OK.

Opinions anyone?

[Bittybye]

I just received my Kobo Aura HD and updated Calibre but every time I add a book Calibre freezes for a few seconds. I'm on a windows 7 laptop

[davidfor]

Is this when you send the book to the AuraHD? How soon after starting the send? What happens when you copy a book to the device without calibre?

[jackie_w]

Originally Posted by davidfor

What is the save template you are using? There is something that happens with a save template that includes series info and books that don't have series. But, I haven't seen it set the series on the device.

My SaveTemplate does include Series info

Books/{author_sort} - {series}{series_index:0>2s| | - }{title}

I also have a metadata plugboard remapping Author in case that's relevant.

Could you run a debug session? Send a book without series info, disconnect, set the debug title in the driver options to the book name, restart in debug mode and reconnect. Then send me the log.

OK, but I'll remove the save template/plugboard variables first and see how that goes.

Do these have multiple authors?

Neither of the 'no longer detected' books have multiple authors. Only one of my current books on the Glo has multiple authors but it has no series and does have the 'series set to first char of title' problem.

What I can do is to change the "Set series info" option to "Set other metadata". Then have it set the series info, publisher, published date and ISBN ... Opinions anyone?

That would be nice to have but don't put it high on your to-do list on my account.

Here is some extra info:
All the books are standard epub with no DRM.

In case my problems are self-inflicted perhaps this is also relevant. Whilst the normal workflow would be:

1. Connect and send-to-disk
2. Disconnect and let reader process new books.
3. Repeat connect/disconnect

I have been adding a semi-automated process (not via calibre code) between 1 and 2. For each newly-sent book on the Kobo disk

- open epub
- append kobo-specific css to existing css file
- rebuild epub in situ

Both file date/time and filesize will therefore increase slightly. Understanding the logic of how things work, can you see why this might cause problems? I would add that I have been doing this for months on fw v2.1.5 with no apparent problems.

I also tried adding my extra process after connecting for a 3rd time, but on disconnect the Glo deleted all updated epubs. This method also used to work OK on v2.1.5

I don't expect the Kobo driver to be changed to accommodate my niche requirements but some understanding of problems I may be creating for myself would be appreciated. I can probably find a workaround if I have to.

I'll send the logs later today.