Kobo book management site (highlights/progress/annotations)

[ndriver]

After posting my last message, I also gave it a second look and found the content table. I also saw the information was there, ready to use...
Though for some reason my content table is really small, it has only the data for the last 30 days or so. Maybe I did something in the past that wiped it out.

With the hard part out-of-the-way and after Quoth's minimalist post questioning why not simply use Calibre, I decided to do just that.

In order to make all this work wirelessly, I use bash scripts that run in the Kobo device, which communicates through SSH to my little linux server that hosts Calibre.
The scripts are called by using the excellent NickelMenu, so everything is well-integrated in the Kobo main menu.

Next weekend I will try to implement some simple graphs, which will show how much was read in a specific week/month etc. These will have to be hosted outside Calibre of course.

The docker suggestion was so that everyone could deploy locally, in a safe and easy way, a SSH server and some web server.

But on second thought, I guess only the maniac people who are interested in stats will bother with this. A container is probably over complicating things for such small gains....

[rigogsilva]

Originally Posted by ndriver

After posting my last message, I also gave it a second look and found the content table. I also saw the information was there, ready to use...
Though for some reason my content table is really small, it has only the data for the last 30 days or so. Maybe I did something in the past that wiped it out.

With the hard part out-of-the-way and after Quoth's minimalist post questioning why not simply use Calibre, I decided to do just that.

In order to make all this work wirelessly, I use bash scripts that run in the Kobo device, which communicates through SSH to my little linux server that hosts Calibre.
The scripts are called by using the excellent NickelMenu, so everything is well-integrated in the Kobo main menu.

Next weekend I will try to implement some simple graphs, which will show how much was read in a specific week/month etc. These will have to be hosted outside Calibre of course.

The docker suggestion was so that everyone could deploy locally, in a safe and easy way, a SSH server and some web server.

But on second thought, I guess only the maniac people who are interested in stats will bother with this. A container is probably over complicating things for such small gains....

About the content table being small, I wonder if kobo does some housekeeping in the sqlite db after a while.

I am curious about the shell script. When using the script, do you trigger it from the Kobo through the menu? (that is awesome!). So, technically I could do the same but point to my cloud server instead (cool). I am looking at the NickelMenu tool. Thanks!

I mean, the data doesn't have to be pushed to my server. I could definitely put it out there as a local web app and people can use locally (saves me some cloud money).

I will take a closer look at the calibre plugin.

[ndriver]

Yes, you can easily call scripts by using the menu.

There are some wizards in this forum who provide us these awesome tools.

Until recently I was using kfmon to run scripts, now I am using NickelMenu, which is even better and easier to use.

Check this post:

https://www.mobileread.com/forums/sh...d.php?t=329525

[davidfor]

Originally Posted by rigogsilva

About the content table being small, I wonder if kobo does some housekeeping in the sqlite db after a while.

The content table contains at a record for each book known to the device, plus a record for each ToC entry for books actually on the device, and an extra record per internal file for kepubs. "Known to the device" is sideloaded books, purchased books (either downloaded or not), preview and recommendations. The latter are cleaned up frequently when the device is synced. If the table is small, it means there aren't many books on the device.

[ndriver]

Originally Posted by davidfor

The content table contains at a record for each book known to the device, plus a record for each ToC entry for books actually on the device, and an extra record per internal file for kepubs. "Known to the device" is sideloaded books, purchased books (either downloaded or not), preview and recommendations. The latter are cleaned up frequently when the device is synced. If the table is small, it means there aren't many books on the device.

I described my situation in a misleading way.
I meant the query of the content table after filtering the results of the ReadStatus column to either 1 or 2 (reading and finished).
There are books I have finished a year ago, which report the correct time. Others finished very recently also seem to be correct. But some books between these periods either disappeared or are reporting a very low value.

I had to repair my Kobo account at some point in the past, because there were a lot of missing covers. And I remember that it had to download a lot of books that were already in my device. Maybe it is related to that.

[davidfor]

Originally Posted by ndriver

I described my situation in a misleading way.
I meant the query of the content table after filtering the results of the ReadStatus column to either 1 or 2 (reading and finished).
There are books I have finished a year ago, which report the correct time. Others finished very recently also seem to be correct. But some books between these periods either disappeared or are reporting a very low value.

I had to repair my Kobo account at some point in the past, because there were a lot of missing covers. And I remember that it had to download a lot of books that were already in my device. Maybe it is related to that.

The reading time is only there for books actually read on the device. For purchased books, it isn't synced in any way. If you read on multiple devices or the Kobo apps, each place only has the time spent reading there. And if you reset the database on the device (sign out or factory reset), you will lose the time spent.

I'm not sure what the repair will do. It probably depends on whether it has to recreate the database entries. I think it checks everything and then replaces anything that is questionable. The latter would probably mean the reading time and related stats would be lost.

[ndriver]

Just for fun, I proceeded with the test of trying to implement a solution to keep an eye on and plot my reading patterns.

This is self hosted. The e-reader sends the result of a query to my PC, where it is parsed.
(A shared site would not be something I would use because of the privacy issues that are inherent to that type of setup.)

I can then lookup and search the results in a web page (though at the moment it is in need of a bit of styling...). After selecting the date range, a stacked bar chart is displayed, showing what and how much was read on a daily, weekly or monthly scale.

Main drawbacks are that ideally the script which sends the query result to the PC should be run everyday the Kobo is used and secondly the results are only valid in scenarios where you use exclusively one device, since this data is not synced to the Kobo servers as davidfor warned on the previous post.

[NiLuJe]

Oh, hai, KOReader feature ^^.

[ndriver]

Originally Posted by NiLuJe

Oh, hai, KOReader feature ^^.

Indeed, I ought to try KoReader again, I haven't done so in ages...

What made me return to Nickel, if I remember correctly, was Pocket, yeah I Know...It is closed source and there is also Wallabag, but I'm so used to Pocket that I just wanted it back.

Would you point me in the right direction regarding the execution of my little script over here?
I would like to trigger the script just before Nickel is suspended. Is there any hook available to make that possible?
Thanks for your great work.

[NiLuJe]

Not yet. Sherman is working on something that may help, but keep in mind that when a device goes to sleep, it goes to *suspend*, so that's generally a terrible trigger to do stuff, because you can't actually do stuff ;p.

[sherman]

Originally Posted by NiLuJe

Not yet. Sherman is working on something that may help, but keep in mind that when a device goes to sleep, it goes to *suspend*, so that's generally a terrible trigger to do stuff, because you can't actually do stuff ;p.

I haven't tested any of these yet, but Nickel does emit the following signals:

SIGNAL :: void N3PowerWorkflowManager::aboutToSleep()
SIGNAL :: void N3PowerWorkflowManager::resumingFromSleep()
SIGNAL :: void N3PowerWorkflowManager::resumedFromSleep()
SIGNAL :: void N3PowerWorkflowManager::aboutToPowerOff()
SIGNAL :: void N3PowerWorkflowManager::aboutToReboot()

So I think it will be possible to listen for them using my current project. Just got to plumb them in...

[rigogsilva]

Originally Posted by ndriver

Just for fun, I proceeded with the test of trying to implement a solution to keep an eye on and plot my reading patterns.

This is self hosted. The e-reader sends the result of a query to my PC, where it is parsed.
(A shared site would not be something I would use because of the privacy issues that are inherent to that type of setup.)

I can then lookup and search the results in a web page (though at the moment it is in need of a bit of styling...). After selecting the date range, a stacked bar chart is displayed, showing what and how much was read on a daily, weekly or monthly scale.

Main drawbacks are that ideally the script which sends the query result to the PC should be run everyday the Kobo is used and secondly the results are only valid in scenarios where you use exclusively one device, since this data is not synced to the Kobo servers as davidfor warned on the previous post.

Ndriver, that is pretty cool. That exactly what I am kind of looking for.

After all the responses about private concerns for sharing data, I have decided to create something different than a website where I would need to keep users' data.

I am thinking about creating a:
- python pip library that will easily export the data out of kobo. I it will also get the page turns out of it since that is encoded. The pip library will decode.
- support exporting data from multiple KoboReader.db files. This will be great if you have backups.
- this pip library will also allow us to run notebooks in a docker container. Notebooks will provide very detail information about your reading. There are so many plots I want to do, the one you have in that picture is one of them
- all data will be kept locally so no sharing of data going on. Keeping your data and having it exported from the device so you can save it.

Maybe one day, I can use ndriver script to automate exporting the data automatically from kobo without a computer.

[rigogsilva]

Originally Posted by NiLuJe

Oh, hai, KOReader feature ^^.

Tried Koreader. That is pretty good.

[MarkCanada]

Originally Posted by ndriver

Just for fun, I proceeded with the test of trying to implement a solution to keep an eye on and plot my reading patterns.

This is self hosted. The e-reader sends the result of a query to my PC, where it is parsed.
(A shared site would not be something I would use because of the privacy issues that are inherent to that type of setup.)

I can then lookup and search the results in a web page (though at the moment it is in need of a bit of styling...). After selecting the date range, a stacked bar chart is displayed, showing what and how much was read on a daily, weekly or monthly scale.

Main drawbacks are that ideally the script which sends the query result to the PC should be run everyday the Kobo is used and secondly the results are only valid in scenarios where you use exclusively one device, since this data is not synced to the Kobo servers as davidfor warned on the previous post.

@ndriver - I know it's been two years, but I just saw your image from post 22 and that's what I am looking for.
Did you ever go anywhere with your stats in a way that it is released or something you can share?