Exporting annotations, bookmarks and reading progress to the cloud

[qkqw]

As I've lost my annotations and reading progress too many times (mostly due to playing around with scripts, installing stuff, automatic import etc.), and I don't use Calibre, I've hacked together a small script that exports my annotations, bookmarks and reading progress to Markdown files.

Currently I'm using it via NickelMenu like this:

menu_item :main    :Sync Annotations    :nickel_wifi     :autoconnect
  chain_success                         :cmd_spawn       :quiet:/mnt/onboard/.adds/notes/notes.sh
  chain_success                         :dbg_toast       :Syncing Notes

I've also added a sqlite3 binary to extract all the information I need from the Kobo database.

If you want to use it, extract the zip file to /mnt/onboard/.adds/notes/ and adjust the directory you want your notes exported to.



Feedback welcome!

Changelog:

2022-10-13:

• Initial release

2022-10-14:

• Fix libsqlite.so.0 typo
• Export file to /mnt/onboard/Exported Annotations/Notes.md
• Add current date/time to the file
• Only show not hidden bookmarks
• Show chapter in highlights
• Show chapter in dogear bookmarks

2022-10-15:

• Combine highlights and dogears to a single list
• Add created date to each highlight
• Add an icon showing whether an entry is a dogear, note or highlight
• Fix compatibility with epub highlights
• Add note regarding export location when not using English as default language

2022-10-16:

• Change document icon in order to support Android
• Add current chapter name to current reads
• Add newlines between headings and quotes (not required by Markdown, but by pandoc)

2022-10-17:

• Order bookmarks by title, chapter, chapter progress and then date
• Simplify upload filename to "notes-%Y-%m-%d.md"

2022-10-17:

• Removed the dependency to KoboCloud
• Removed the Nextcloud Curl command
• Exported notes are now named with their current date (e.g. notes-2023-02-17.md)
• Notes are rotated and 21 notes are kept (can be configured)

2023-02-17:

• Remove KoboCloud dependency
• Rename notes with their current date

2023-08-04:

• Export book progress of PDFs as well

2023-10-16:

• Fix a bug in reading progress that didn't list a book when it doesn't have any chapter titles (e.g. PDFs or automatically generated Epubs)

2024-03-26:

• Simplify SQL query and don't exclude bookmarks if Epub is partially broken or has no chapter titles

[mtxo]

Hello. I just register me to reply to this post.

Many thanks for this script, it works well for me. But I don't use it for store annotations in the cloud. I have modify a bit the script and put that it copies de results to the Kobo itself, and them I save it to my PC through the usb cable.

It is more powerful that the Kobo system, that lets to save the annotations of only one book at once!!

But there is a bug in the zip. The archive in the lib folder need to name it as libsqlite3.so.0 (see the .0 at the end), and no as libsqlite3.so like in the zip file. With this change, the script is working well now!!

Many thanks for it. Excuse my bad english. Greetings. Mtxo.

[qkqw]

Thanks for your feedback! I've indeed fixed the libsqlite3.so.0 typo on my Kobo, but not on my system I've just uploaded a new version with some more features and improvements, e.g. showing the current chapter and exporting the notes to /mnt/onboard/Exported Annotations/Notes.md instead of a temporary file.

If you want to share your version or any other improvements, I'd be happy to integrate them!

[mtxo]

Many thanks for the improvements.

I not interested in the cloud part of the script, so as now it saves the markdown file in the kobo storage, it is so fine for me.

In the first script I was made this "improvement":

#$CURL -u "$USER" -H "$HEADER" -T "$TMP" "$URL/notes-$DATE.md"
cp "$TMP" /mnt/onboard/mynotes-$DATE.md

So: comment the line that make the cloud connection, and put a new line for save temp file to the kobo storage. So, nothing interesting

I'm not a coder, so I can't improve your code

I test the new version. The new code for save the markdown file to the kobo storage is ok. But, maybe it have a "bug" in the case of a non english environment. Mine is spanish, and the "Exported Annotations" dir is called "Anotaciones exportadas" in spanish. So maybe you need to put a note in the script to the user to adjust it to her language...

And the result markdown file is not "the same" in the two versions. In the previous version I have a more complete file, whit probably all the highlights of the kobo saved. In the new one, there are not highlights for some books. You can see them in the attached files (i renamed them to the .txt extension). The 1.txt file is made with the first version, and the 2.txt file with the second.

As you can see, in 1.txt there are highlights from these books:
- Autonomous, Annalee Newitz
- Informe sobre Biodiversidad (a lot of them)
- Instrucciones para salvar el mundo, Rosa Montero
- La libertad interminable, Joe Haldeman
- Panteón, Carlos Sisí

In the second file there are only highlights from "Informe sobre Biodiversidad". There are all of them, but in another order. Maybe, the order of these highlights in the first file is better, but I not sure.

If you need my kobo database, I can send you, but I prefer to not make it public, so I will send to you only.

That's all. Many thanks for you work. Greetings. Mtxo.

[qkqw]

Originally Posted by mtxo

I test the new version. The new code for save the markdown file to the kobo storage is ok. But, maybe it have a "bug" in the case of a non english environment. Mine is spanish, and the "Exported Annotations" dir is called "Anotaciones exportadas" in spanish. So maybe you need to put a note in the script to the user to adjust it to her language...

Oh I didn't know that, I always assumed it would be the same name. But I'll add a note.

Originally Posted by mtxo

And the result markdown file is not "the same" in the two versions. In the previous version I have a more complete file, whit probably all the highlights of the kobo saved. In the new one, there are not highlights for some books. You can see them in the attached files (i renamed them to the .txt extension). The 1.txt file is made with the first version, and the 2.txt file with the second.

Do you use epub or kepub books?

Also, could you try to remove the line

AND b.Hidden = 'false'

?

It seems like the dogears work fine, but only the highlights section is not complete.

[mtxo]

I normally use epub format, but in some cases I convert it to kepub, mostly for see the footnotes. But all the books that have highlights now are epubs (in the kobo storage I have some kepubs, but they don't have highlights).

I test the proposed change. I first remove that line in the SQL code for the highlights, and make a probe. The result is the same, no changes. Later I remove too the same line in the SQL for the dogears, and nothing changes, the resulted file is the same that in the first probe, and is the same that in the probes for yesterday.

[mtxo]

Only for completeness. The book that have the most highlights, "Informe sobre Biodiversidad", was a long M$ Word document, that I convert it to epub with Libreoffice. Later sideloaded to the kobo for carefully read it, make annotations, and export them with the Kobo's own mechanism.

[qkqw]

Originally Posted by mtxo

I normally use epub format, but in some cases I convert it to kepub, mostly for see the footnotes. But all the books that have highlights now are epubs (in the kobo storage I have some kepubs, but they don't have highlights).

Thanks for pointing this out. I almost exclusively use kepub, so I didn't notice that epub uses a slightly different format for storing its chapters in the database. I've (hopefully) fixed this with a new version.

Want to try and see if it works for you?

[mtxo]

The new code works sooo well!!!

I get a file like the made with the first version, with all the highlights, and all the dogears than in the first version, with the same order too. And now (I don't know why) I have a "new" dogear that was in a PDF file, that did not appear in the first version!!

And I get too the new improvements: the chapters, the date and hour of the highlights, the icons for the highlights and dogears. I think that I have all of them!!

Attached you can see a screenshot of part of the new file, that I convert to html with pandoc, in Debian linux. You can see in it the new dogear from the pdf file.

Many thanks for all the work. It may be useful. Greetings. Mtxo.

[qkqw]

Originally Posted by mtxo

The new code works sooo well!!!

You are welcome

I've just pushed a few minor changes, like showing the chapter name in current reads and adding a few newlines in order to have pandoc render the Markdown correctly.

[vinnyt]

Thanks for creating this. This is great!

I tested it out as well and modified it a bit to upload to my local NAS that hosts a webdav server. Everything is working however the order by statement might need a fix.

I'm thinking that it should be sorted in the following order: Book -> chapter -> pageNum instead of dateCreated in the final sort of the query as it is now. Not sure if you have access to the page number but to me it makes more sense to have the annotations in order by page number within a chapter.

[vinnyt]

I may have figured out the issue mentioned above. Chapter progress seems to do the trick.

ORDER BY c.BookTitle ASC,
c.VolumeIndex ASC,
b.ChapterProgress ASC;"

[qkqw]

Originally Posted by vinnyt

Thanks for creating this. This is great!

Thanks

Originally Posted by vinnyt

I tested it out as well and modified it a bit to upload to my local NAS that hosts a webdav server.

Happy to integrate any changes or make it easier to support other webdav/upload solutions.

Originally Posted by vinnyt

I'm thinking that it should be sorted in the following order: Book -> chapter -> pageNum instead of dateCreated in the final sort of the query as it is now. Not sure if you have access to the page number but to me it makes more sense to have the annotations in order by page number within a chapter.

I thought it make sense to order by the creation of a bookmark instead of the chapter progress, as that's my reading style. But I don't mind changing this, see the new version.

I'd love to have access to the page number. But it seems that page numbers will be calculated at runtime, depending on your current font and margin settings.

[vinnyt]

Looking much better. However I'm still seeing an issue with dogears bubbling to the top of the output even though it belongs to a later chapter/subchapter. Possibly a bug in the query for chapter identification for dogears.

Also here is an updated config script. I added a 'sleep 5' just before the http call in the notes.sh script. This is to make sure there is enough time for wifi to connect. The last curl command returns an error and the main NM config waits upto to 10 seconds for everything to complete.

menu_item :main :Sync Annotations :nickel_wifi :autoconnect
chain_success :cmd_output :9999:quiet:/mnt/onboard/.adds/notes/notes.sh
chain_success:skip:2
chain_failure :dbg_toast :Failed to sync notes!
chain_always:skip:-1
chain_success :dbg_toast :Notes Sync Complete!

[qkqw]

Originally Posted by vinnyt

Looking much better. However I'm still seeing an issue with dogears bubbling to the top of the output even though it belongs to a later chapter/subchapter. Possibly a bug in the query for chapter identification for dogears.

It looks fine with my books. Does it happen for all your books or just a single one?

Originally Posted by vinnyt

Also here is an updated config script. I added a 'sleep 5' just before the http call in the notes.sh script. This is to make sure there is enough time for wifi to connect. The last curl command returns an error and the main NM config waits upto to 10 seconds for everything to complete.

Ideally we'd have to use a ping test or similar to find out if an connection is working (e.g. https://github.com/fsantini/KoboClou...get.sh#L21-L37).

The sleep however doesn't have to be included in the script, as NickelMenu supports that too:

menu_item :main :Sync Annotations :nickel_wifi :autoconnect
chain_success :cmd_output :9999:quiet:sleep 5
chain_success :cmd_output :9999:quiet:/mnt/onboard/.adds/notes/notes.sh
chain_success:skip:2
chain_failure :dbg_toast :Failed to sync notes!
chain_always:skip:-1
chain_success :dbg_toast :Notes Sync Complete!

You might also want to hook this script into KoboCloud or run it manually, hence a sleep 5 inside the script is IMHO not the best approach.