How to programmatically insert book highlights imported from Apple Books?

[wealthychef]

Hello
I have a list of highlights from Apple Books app, and am trying to move them all to an open format.
The opf file in the Calibre directories looked promising but I'm told these files are not the Way as it is only used to restore the database, which never gets corrupted (q.v. https://www.mobileread.com/forums/sh...15&postcount=5). Is there a way to use a shell script and cram my highlights into Calibre from the command line? :-). Thanks
And conversely can they be easily exported? My goal here is to own my own data.
Thank you!

[DNSB]

The first question that comes to mind is what format are the exported hightlights in?

[kovidgoyal]

If you want to export calibre highlights you can do so in multiple formats including markdown and json. As for importing highlights, you would first need to write code to convert the highlights from whatever format they are to the format calibre uses, which will be a pretty non-trivial exercise.

[wealthychef]

Originally Posted by DNSB

The first question that comes to mind is what format are the exported hightlights in?

A very simple one, with just each line containing a highlight, a tag for the color, and a date. Apple does not give much information. I'm using Readwise at the moment until I figure a better way. It's really a mess.

I think I just want a spec for the format, the conversion is probably not that difficult using something like python. It seems to me that having my highlights in Calibre will allow a more rich exploration and integration with my other tools than Apple's proprietary approach.

[wealthychef]

Originally Posted by kovidgoyal

If you want to export calibre highlights you can do so in multiple formats including markdown and json. As for importing highlights, you would first need to write code to convert the highlights from whatever format they are to the format calibre uses, which will be a pretty non-trivial exercise.

Is the format really that complex? I was thinking to just spit out lines into a file using python. I've done file format conversions before, and it's just a matter of knowing the spec in my experience.

My thinking for storing the highlights in Calibre instead of Apple Books are:
1. I want to own my data
2. I want to use the data in other tools like Obsidian.md and bash scripts

The hard part is figuring out page numbers or file locations for the highlights, because Apple does not store that(!)

[davidfor]

Originally Posted by wealthychef

A very simple one, with just each line containing a highlight, a tag for the color, and a date. Apple does not give much information. I'm using Readwise at the moment until I figure a better way. It's really a mess.

I think I just want a spec for the format, the conversion is probably not that difficult using something like python. It seems to me that having my highlights in Calibre will allow a more rich exploration and integration with my other tools than Apple's proprietary approach.

Originally Posted by wealthychef

Is the format really that complex? I was thinking to just spit out lines into a file using python. I've done file format conversions before, and it's just a matter of knowing the spec in my experience.

My thinking for storing the highlights in Calibre instead of Apple Books are:
1. I want to own my data
2. I want to use the data in other tools like Obsidian.md and bash scripts

The hard part is figuring out page numbers or file locations for the highlights, because Apple does not store that(!)

The complexity is if you want the annotation located in the book. The location of the annotation used by the other system has to be converted to that used by calibre. And that might not be easy. It gets harder if the books are not the same version (different formatting). And if there format differences, it gets even harder.

For what you want, the Annotations plugin possibly would work. This import annotations from some devices and export files, builds a bit of HTML and stores it in a column in calibre against a book. It doesn't attempt to match the the location of the annotations within the book, just shows them with the information available. The plugin is extendible with reader code for the device or export format. It should be fairly simple for what you want. The complexity in the code tends to be when matching the book on the device to books in the library, but, the format you describe sounds like the file is for a single book.