[Conversion Plugin] KFX Input

[willemml]

Originally Posted by jhowell

The notebook folder name is based on the metadata of the KFX book being annotated. It is composed of the content_id, cde_content_type, and the string "notebook"; all separated by "!!". For example "EBEA035E6DB444159EF42DA7E5EEF8F6!!PDOC!!notebook" .

This much I had mostly figured out myself, but thank you for confirming and giving me the correct names for each part.

Originally Posted by jhowell

The EPUB produced by KFX Input from a Scribe annotation notebook contains one XHTML file per annotation, each linking to an SVG image. The connection between a book page (in KFX format produced from PDF) and the associated annotation notebook page is provided by a file with the extension .yjr found in the .sdr folder associated with the KFX book. That file can be converted to JSON using KRDS - A parser for Kindle reader data store files.

Each annotated page will have an entry such as:

    "annotation.cache.object": {
        "annotation.personal.handwritten_note": [
            {
                "startPosition": "201.0:13974",
                "endPosition": "201.0:13974",
                "creationTime": "2023-08-26T09:09:38.130000",
                "lastModificationTime": "2023-08-26T09:09:38.130000",
                "template": "0\ufffc0",
                "handwritten_note_nbk_ref": "crEq-GhRTSa63nk5j3KC6Qw0"
            }
        ]
    },

The startPosition is a KFX position number that corresponds to the book page being annotated. The page number can be found by looking up the part of the position number following the colon in a content JSON file that can be optionally produced by the CLI of the KFX Input plugin. (The number will match a type 2 entry. Count type 2 entries in the file to find the page number.)

Ah, wonderful, this is exactly what I was looking for. Thank you. Good to know that pages are type 2 entries, I had vaguely determined this already from the JSON output of this plugin, but was not sure. Do you know what all the content types are? I have so far only come across content type 2, but I guess that is because currently all the files I am examining are from print replica books created from PDF files.

Originally Posted by jhowell

The handwritten_note_nbk_ref is the KFX section ID of the associated annotation page in the notebook. Currently those IDs are not reflected in the EPUB generated by the KFX Input plugin for an annotation notebook. I will update the plugin to include this data in the EPUB so that these can be matched.

Thank you, that will help a lot with my scripting.

Originally Posted by jhowell

The margins of the PDF page may be been trimmed during conversion to KFX format for delivery to the Scribe. Also the SVG produced will have the aspect ratio of the Scribe screen which might not match the PDF page. Because of this some image manipulation may be needed to properly overlay the SVG image onto the original PDF page.

If I were to extract the PDF from the KFX files I am generating I assume that would save me from dealing with trimming the PDFs for correct alignment? And is there a point I can align the SVG to (say top left or right or similar) of the PDF with an offset to account for aspect ratio change?

In the mean time I am trying to write a program that can convert from PDFs to write-on-able KFX files without going through the Kindle Create software (which for now means I am trying to create my own KPF files from scratch that contain the metadata to correctly map PDF pages) so that I can put them through the KFX Output plugin (which I realize still relies on Kindle Previewer for conversion, but I eventually want to write my own KPF to KFX converter as well.) I will be publishing all code on GitHub as soon as I have something that works a little bit. Is there any documentation outside of this thread and the code of the KFX plugins you have (which is the closest I could find to documentation on the KFX and KPF formats other than a rough overview) that could help me?

[jhowell]

Originally Posted by willemml

Good to know that pages are type 2 entries, I had vaguely determined this already from the JSON output of this plugin, but was not sure. Do you know what all the content types are? I have so far only come across content type 2, but I guess that is because currently all the files I am examining are from print replica books created from PDF files.

Type 1 is text, type 2 is an image. For a print replica KFX there is a single image associated with each PDF page. There may also be type 1 entries if the source PDF had a text layer that was included in the KFX version of the book.

Originally Posted by willemml

If I were to extract the PDF from the KFX files I am generating I assume that would save me from dealing with trimming the PDFs for correct alignment?

As far as I can tell the included PDF is essentially the same as the original PDF, at least in terms of its page dimensions. The four margins for cropping are instead expressed as KFX metadata associated with each PDF page image resource.

That cropping data is currently not handled by the KFX Input plugin. Something will need to be done about that.

Originally Posted by willemml

And is there a point I can align the SVG to (say top left or right or similar) of the PDF with an offset to account for aspect ratio change?

Each XHTML page of the EPUB converted from the annotation notebook has an embedded SVG image that references the external SVG image containing the stroke data. If the aspect ratio of the annotation does not match the PDF then the style associated with the embedded SVG will have the height, width, top, and left properties needed to adjust it to match the associated PDF page aspect ratio.

Originally Posted by willemml

In the mean time I am trying to write a program that can convert from PDFs to write-on-able KFX files without going through the Kindle Create software (which for now means I am trying to create my own KPF files from scratch that contain the metadata to correctly map PDF pages) so that I can put them through the KFX Output plugin (which I realize still relies on Kindle Previewer for conversion, but I eventually want to write my own KPF to KFX converter as well.) I will be publishing all code on GitHub as soon as I have something that works a little bit.

I am curious to see what you come up with.

Originally Posted by willemml

Is there any documentation outside of this thread and the code of the KFX plugins you have (which is the closest I could find to documentation on the KFX and KPF formats other than a rough overview) that could help me?

I do not know of any existing documentation other than that provided by Amazon for the underlying Ion data format.

[willemml]

Originally Posted by jhowell

Type 1 is text, type 2 is an image. For a print replica KFX there is a single image associated with each PDF page. There may also be type 1 entries if the source PDF had a text layer that was included in the KFX version of the book.

Great thanks, that's good to know.

Originally Posted by jhowell

As far as I can tell the included PDF is essentially the same as the original PDF, at least in terms of its page dimensions. The four margins for cropping are instead expressed as KFX metadata associated with each PDF page image resource.

That cropping data is currently not handled by the KFX Input plugin. Something will need to be done about that.

Each XHTML page of the EPUB converted from the annotation notebook has an embedded SVG image that references the external SVG image containing the stroke data. If the aspect ratio of the annotation does not match the PDF then the style associated with the embedded SVG will have the height, width, top, and left properties needed to adjust it to match the associated PDF page aspect ratio.

That simplifies things for me in the end I think, the cropping metadata should give me enough to achieve a correct overlap and lets me keep the original PDF sizing (this is desirable as I would be sending PDFs to my Kindle and then taking them back and uploading them somewhere in an annotated form.) And really helpful that the SVG style includes the adjustment properties.

Originally Posted by jhowell

I am curious to see what you come up with.

I will be posting very soon, I have a mostly working serializer for KRDS files (I know this might sound pointless, and I know there is already a Python deserializer) but I want to have something that can go both ways and in a strongly typed and compiled language (Rust in this case). I am also using this as a sort of warm up project to serializing and deserializing KDF, KPF and KFX files. I also have hopes that I might be able to create annotations on my computer and send them to the kindle in a way that it can recognize them (rather than just hardcoding them into a PDF or book.)

[jhowell]

Originally Posted by willemml

That simplifies things for me in the end I think, the cropping metadata should give me enough to achieve a correct overlap and lets me keep the original PDF sizing (this is desirable as I would be sending PDFs to my Kindle and then taking them back and uploading them somewhere in an annotated form.) And really helpful that the SVG style includes the adjustment properties.

I did some further research and found that the cropping data contained in the book being annotated is not needed for overlaying annotations on pages. The SVG style (width, height, top, and left) has all of the information that is needed to adjust the annotation rectangle to fit over the uncropped PDF page.

One thing to be aware of is that due to different aspect ratios between the scribe screen and cropped PDF pages there are typically parts of the page in the margins that cannot be annotated because they are off screen. Alternatively there can be places on the screen where annotations can be drawn that will be outside of the page boundaries of the original PDF.

Originally Posted by willemml

I also have hopes that I might be able to create annotations on my computer and send them to the kindle in a way that it can recognize them (rather than just hardcoding them into a PDF or book.)

I look forward to seeing what you come up with.

[jhowell]

Version 2.4.0 - 01 Sep 2023

Use the KFX section name as the XHTML and SVG file names in the EPUB produced when converting Scribe notebook pages. This will make it possible to associate handwritten annotations with the book content being annotated using information from the .yjr file of the book.

Ignore page cropping data found in some print replica books. (Fixes "ERROR: resource xx has extra data: {$48: 160.0, $49: 137.0, $50: 160.0, $47: 137.0}")

[tomsem]

Cool. Looking forward to what @willeml comes up with.

[jhowell]

Version 2.5.0 - 08 Sep 2023

Fix bug introduced in previous plugin release for some print replica books. (Fixes "ERROR: Exception checking book consistency: KeyError('$564')")

Fix handling of print replica books that reuse PDF pages.

Handle a variant of hero image formatting used in some KFX books. (Fixes "ERROR: Unexpected combination of alpha (0.1796875) and opacity (1) for color rgba(255,255,255,0.180)")

Fix incorrect formatting applied to headings for some books.

Allow small variations in brush sizes in strokes contained in Scribe notebooks.

[yuhfrfr]

@jhowell So I'm able to import/DeDRM a KFX-zip file that contains guided view from Comics and if I side load to my iPhone it works perfectly. Just curious, outside of KFX, is there any other way the Kindle iOS/Android app will recognize guided view? I read earlier that you could convert kfx to epub, import the epub to Kindle Previewer and then export the azw3 from previewer and that would also have guided view, but only for hardware e-readers?

Also, I currently run the cli to convert KFX to epub, then tried to convert that epub to ZIP and rename to CBZ but the comic pages are all out of order, like page 2 is actually page 6 or something. epub looks fine though. Any idea how I could get the right page order for cbz?

Thanks a lot for your work in all these plugins btw, its great to have guided view with my backups

[jhowell]

Originally Posted by yuhfrfr

Just curious, outside of KFX, is there any other way the Kindle iOS/Android app will recognize guided view? I read earlier that you could convert kfx to epub, import the epub to Kindle Previewer and then export the azw3 from previewer and that would also have guided view, but only for hardware e-readers?

Exporting in .mobi format from Kindle Previewer should work for the Kindle Android app. For the iOS app you need to change the file type during Kindle Previewer export to .azk from the default .mobi and sideload that file instead.

Originally Posted by yuhfrfr

Also, I currently run the cli to convert KFX to epub, then tried to convert that epub to ZIP and rename to CBZ but the comic pages are all out of order, like page 2 is actually page 6 or something. epub looks fine though. Any idea how I could get the right page order for cbz?

The image file names used in KFX carry through to the resulting EPUB and are essentially random. CBZ is displayed in order by image file names.

To fix this you can open the EPUB in the calibre ebook editor, select all of the images, right click & choose "Bulk rename the selected files", and in the dialog box that pops up select "Rename files according to their book order".

Originally Posted by yuhfrfr

Thanks a lot for your work in all these plugins btw, its great to have guided view with my backups

I'm glad it is of use.

[neonclash]

I am experiencing some odd behavior from this plugin. When I import a DRM'd kfx-zip, the decryption appears to succeed, but it still packs the added book into another kfx-zip. The file sizes are different, so some change occurred. Inside the new kfx-zip, the "decrypted" .res files are all about 100k smaller, and the .azw is about 3x larger. My calibre-debug output during import is in B0CGLB24FR.txt.

Another oddity happens when I attempt to unpack or convert this kfx-zip to either PDF or epub. I get the output in the B0CGLB24FR_conv.txt attachment (AttributeError: 'NoneType' object has no attribute 'PdfReader'). This issue was said to be fixed with this latest version, but it's possible the input is bad since it didn't convert to a plain kfx. The preview (it denies cloud reader) appears to be print replica, but I don't know for sure.

I can't tell if this is an issue with DeDRM or this plugin, or if these are one issue or two separate ones. But I can say with certainty that my current combination of plugins worked on another new-format DRM (B0xxx ASIN) book. I'd greatly appreciate any assistance with narrowing down the issue.

[anil4523]

I had tried kfx to pdf command line of kfx input plugin for print replica book
after conversion to PDF, pagination and bookmarks of PDF file not present as Compare to Original PDF file

Do we have any solution for it ??

Thanks

[jhowell]

Originally Posted by anil4523

I had tried kfx to pdf command line of kfx input plugin for print replica book
after conversion to PDF, pagination and bookmarks of PDF file not present as Compare to Original PDF file

Do we have any solution for it ??

Thanks

No. Those are not preserved in most cases.

Amazon’s print replica KFX format includes the original PDF from the publisher split into individual PDF files of a few pages each. The plugin recombines them into a single PDF but it does not restore the links that were broken when the original file was split up. Doing that would be a major undertaking.

[Lief]

I just want to thank you for this plugin. Now that amazon is pushing kfx on everything (even manga) and while dedrm still works to remove the kfx drm, this is a great option and it's working great.
Let's hope it will stay this way for a long long time. Of course I'm using a lot more the kobo store, but sometimes I got some amazon gift cards to use and be able to still backup my ebooks is great.

[APGHOSH]

The plugin is working very well in most of the aspects.

[beam05]

I do not understand this at all.

Open a command/terminal window in your OS and change to the folder where the KFX file was saved.
Use "--pdf" option of the KFX Input plugin CLI to extract PDF content. Eg:
calibre-debug -r "KFX Input" -- --pdf "my book file.kfx"

I have a KPF file from Kindle Create that I want to convert to EPUB while keeping all of the format. Can you explain this in details? I'm very new, sorry.