[Conversion Plugin] KFX Input

[jhowell]

Originally Posted by tomsem

Failures: ...

Thanks for the error reports! I will take a look at them today and will let you know if I need any additional information to help solve them.

I apologize for relying on you so much to do testing and provide samples. I have been considering picking up a Scribe for reading and experimentation but have been reluctant because of the cost and the fact that I have no interest in using it for note taking.

The low end Essentials Bundle is $285 right now for Prime members. That is tempting me.

[jhowell]

Originally Posted by tomsem

Key Error

That relates to a feature (local_delta_fragments) used in Scribe notebooks. I made some assumptions about how to handle that based on the samples I had seen, but your new example includes a case that I didn't account for. This will be fixed in the next plugin release, at least for all of the cases I have encountered so far.

Originally Posted by tomsem

Insufficient data

That notebook uses a brush type (value 0 internally) that I have not encountered before. Strokes for that are missing some of the data that is present for other brush types causing the plugin to fail.

I have seen "brush pen" mentioned as a possible brush type in the Scribe software. Is that what you used to draw in this notebook?

Originally Posted by tomsem

Book has short symbol format without document_data max_id (virtually every !!EBOK!! & !!PDOC!! nbk) - I think most of these are 'empty' e.g. no pen annotations

Those errors result from the notebook being empty. That will be fixed in the next plugin release.

Originally Posted by tomsem

Missing sections

Unfortunately I no longer have the PDOC corresponding to the missing_sections nbk (it was orphaned somehow when I deleted the KFX-PR document). I'll see if I can find another example.

Your "repeats previous location" sample also has this error so I do not need a separate example of it.

It is possible that error results from deleted sticky note annotations not being completely removed from the notebook. One way to test this would be to add a sticky note to a book that has not been previously annotated and then delete the note. I suspect that the resulting annotation notebook will exhibit this problem.

If so then I will change this from an error to a more specific warning in the plugin. If that is not the actual cause then more research will be needed.

Originally Posted by tomsem

repeats previous location

This appears to be a relatively new behavior, since sequential points of a stroke never had the same location in any of the Scribe notebook I examined previously. I will handle this case in the next release of the plugin.

[jhowell]

Originally Posted by tomsem

repeats previous location

I have spent several weeks working on and off to understand how the Scribe derives the thickness and density adjustment factors from the more primitive data (x/y coordinates, tilt angle, and pressure applied.) Earlier this week I had a breakthrough and was able to exactly replicate the thickness and density of the hundreds of thousands of points in the Scribe notebook samples I have seen so far.

I did some more testing and that one notebook sample you provided (95E4A4423B184AE1B3DCF747F659B0CD!!PDOC!!notebook) breaks all of those rules. Do you have any idea what might be different about that notebook as compared with all of the other notebooks you have shared so far?

[tomsem]

Originally Posted by jhowell

I have spent several weeks working on and off to understand how the Scribe derives the thickness and density adjustment factors from the more primitive data (x/y coordinates, tilt angle, and pressure applied.) Earlier this week I had a breakthrough and was able to exactly replicate the thickness and density of the hundreds of thousands of points in the Scribe notebook samples I have seen so far.

I did some more testing and that one notebook sample you provided (95E4A4423B184AE1B3DCF747F659B0CD!!PDOC!!notebook) breaks all of those rules. Do you have any idea what might be different about that notebook as compared with all of the other notebooks you have shared so far?

I do have an idea.

The document in this case started life as a PDF containing 8 Sudoku puzzles. I generally 'permanently delete' these as soon as I have solved the puzzles. This removes it from cloud, as well as the KFX, .sdr folder, and (in this case) the associated .notebook folder.

The fact that this folder had been orphaned suggests that it was deleted on my other Scribe. This was probably while I was doing experiments to see how sync of pen annotations worked in various scenarios.

So this document may have been one that I annotated with both Scribes. It may be some situation due to sync sequencing. I remember one case (but maybe not this one) where I had two solutions overlaid, for example.

Or it might be a simpler matter of having done a Delete Page operation along the way, and having things in undo buffer when document was closed?

At any rate, these are some scenarios that might factor in here (I'll try the Delete Page one to see what results)

Note that Permanently Delete does not remove personal documents from all devices, just from the one that initiated it, and from cloud storage. So you need to also Delete from the other device(s). It still seems like this should not orphan anything however.

(A few weeks ago I wrote a Python script to clean up orphaned .sdr folders, empty folders, and content specific temp files. At the time I had not observed any of these in .notebook folder so it does not look for any. Clearly I need to update it now...)

[PoP]

Originally Posted by jhowell

...several weeks...hundreds of thousands of points...

Kudos for a truly remarquable reverse engineering!

[jhowell]

Originally Posted by PoP

Kudos for a truly remarquable reverse engineering!

Thanks. However I wasted a lot more time than I should have trying to find a simple formula that fit the data. In the end I went though the actual Scribe firmware to discover that it uses cubic Bézier curves to map raw data from the pen into the factors that control the thickness and density of strokes.

[jhowell]

Originally Posted by jhowell

That notebook uses a brush type (value 0 internally) that I have not encountered before. Strokes for that are missing some of the data that is present for other brush types causing the plugin to fail.

After further research I found that this is caused by using the "pen" brush type in the original Scribe firmware. It was changed in later firmware. The next plugin update will handle this properly.

Originally Posted by jhowell

I did some more testing and that one notebook sample you provided (95E4A4423B184AE1B3DCF747F659B0CD!!PDOC!!notebook) breaks all of those rules.

This was not as bad as I first thought. After further testing I was able to determine that the strokes in that notebook have a slight jitter applied to the x/y coordinates of points that was throwing off my calculations. I am assuming that this Sudoku puzzle was annotated using a previous Scribe firmware version with that feature. The next plugin update will handle this properly.

-----

At this point I believe that I have accounted for all of the problems in the last set of sample notebooks and will release the updated plugin after a bit of further testing. Thanks again for all of the help.

[jhowell]

Version 2.2.0 - 10 Jul 2023

Miscellaneous improvements in the conversion of Scribe notebooks including handling of empty notebooks, unreferenced pages, original firmware "pen" brush type, repeated point locations, and local_delta_fragments.

[exy]

Conversion fails with following error for one book. is there any solution?
Latest Calibre and plugins on Win.


WARNING: This book contains PDF content, which can be extracted using the KFX Input plugin CLI.
ERROR: This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.
Converting book to EPUB 3
Format is fixed layout
Traceback (most recent call last):
File "calibre_plugins.kfx_input.__init__", line 115, in convert
Exception: This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "runpy.py", line 196, in _run_module_as_main
File "runpy.py", line 86, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 38, in gui_convert_override
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1108, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre_plugins.kfx_input.__init__", line 126, in convert
calibre.ebooks.conversion.ConversionUserFeedBack: {"msg": "<b>Cannot convert ABC - X : Pqr stu vwxyz</b><br><br>Exception('This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.')", "level": "error", "det_msg": "", "title": "KFX conversion failed"}

[jhowell]

Originally Posted by exy

WARNING: This book contains PDF content, which can be extracted using the KFX Input plugin CLI.
ERROR: This book has a layout that is incompatible with calibre conversion. For best results use the KFX Input plugin CLI for conversion.

The calibre conversion system is designed to handle reflowable books. Fixed layout (page layout that cannot be changed) is not supported.

See the Command Line Interface section in the first post of this thread for instructions on using the plugin's CLI to convert fixed layout books.

[jhowell]

Version 2.3.0 - 07 Aug 2023

Handle new heading level metadata. (Fixes "Unexpected Ion symbols used: $798, $799, $800" and "nav_container xxx has unknown type: $798")

Fix error when processing the cover of fixed layout books converted from KPF format. (Fixes "YJFragmentList item is missing: '$389'")

[xvicarious]

This plugin seems to treat KPF and KFX files the same. Say if I have a KPF file, and try to use your other plugin to convert to to KFX, it will create an intermediate EPUB file, while the KFX Output plugin handles KPF files.

[jhowell]

Originally Posted by xvicarious

This plugin seems to treat KPF and KFX files the same. Say if I have a KPF file, and try to use your other plugin to convert to to KFX, it will create an intermediate EPUB file, while the KFX Output plugin handles KPF files.

The plugins have different functionality depending on whether they are used as part of a calibre conversion process or invoked using their command line interfaces. Calibre conversion always uses EPUB as the intermediate format regardless of the selected input and output formats. The plugin CLIs do more direct conversion.

KPF is closely related to KFX. Either can be used as an input format for the KFX Input plugin. And the CLI of KFX Output can transform KPF directly into KFX.

[willemml]

Hello, I am trying to write a script that will let me extract PDFs from the Kindle in their annotated form without contacting amazon. Do the annotation "notebooks" (each write-on PDF on the Kindle seems to have a corresponding notebook folder that when put through KFX Input gives me the SVG of all my annotations.) Do these "notebooks" contain info on which SVG goes with which page of each PDF? (Even if it is not in the corresponding epub file and only tells me which pen traces go on each page in the KFX nbk file.) If so where is this data stored?

[jhowell]

Originally Posted by willemml

Hello, I am trying to write a script that will let me extract PDFs from the Kindle in their annotated form without contacting amazon. Do the annotation "notebooks" (each write-on PDF on the Kindle seems to have a corresponding notebook folder that when put through KFX Input gives me the SVG of all my annotations.)

The notebook folder name is based on the metadata of the KFX book being annotated. It is composed of the content_id, cde_content_type, and the string "notebook"; all separated by "!!". For example "EBEA035E6DB444159EF42DA7E5EEF8F6!!PDOC!!notebook" .

Originally Posted by willemml

Do these "notebooks" contain info on which SVG goes with which page of each PDF? (Even if it is not in the corresponding epub file and only tells me which pen traces go on each page in the KFX nbk file.) If so where is this data stored?

The EPUB produced by KFX Input from a Scribe annotation notebook contains one XHTML file per annotation, each linking to an SVG image. The connection between a book page (in KFX format produced from PDF) and the associated annotation notebook page is provided by a file with the extension .yjr found in the .sdr folder associated with the KFX book. That file can be converted to JSON using KRDS - A parser for Kindle reader data store files.

Each annotated page will have an entry such as:

    "annotation.cache.object": {
        "annotation.personal.handwritten_note": [
            {
                "startPosition": "201.0:13974",
                "endPosition": "201.0:13974",
                "creationTime": "2023-08-26T09:09:38.130000",
                "lastModificationTime": "2023-08-26T09:09:38.130000",
                "template": "0\ufffc0",
                "handwritten_note_nbk_ref": "crEq-GhRTSa63nk5j3KC6Qw0"
            }
        ]
    },

The startPosition is a KFX position number that corresponds to the book page being annotated. The page number can be found by looking up the part of the position number following the colon in a content JSON file that can be optionally produced by the CLI of the KFX Input plugin. (The number will match a type 2 entry. Count type 2 entries in the file to find the page number.)

The handwritten_note_nbk_ref is the KFX section ID of the associated annotation page in the notebook. Currently those IDs are not reflected in the EPUB generated by the KFX Input plugin for an annotation notebook. I will update the plugin to include this data in the EPUB so that these can be matched.

The margins of the PDF page may be been trimmed during conversion to KFX format for delivery to the Scribe. Also the SVG produced will have the aspect ratio of the Scribe screen which might not match the PDF page. Because of this some image manipulation may be needed to properly overlay the SVG image onto the original PDF page.