[Conversion Plugin] KFX Input

[laxarus]

Updated to the 2.13.0 and I started getting this error.

Even the books I was able to convert from KFX to epub without problem started giving this error.
What is wrong here?
I am trying to find an older version of the KFX Input to test but cannot find it anywhere.

calibre, version 7.12.0
ERROR: Unhandled exception: <b>ConstraintError</b>:ConstraintError: Foreign key violation: tags is still referenced

calibre 7.12 embedded-python: True
Windows-10-10.0.22631-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22631')
Python 3.11.5
Windows: ('10', '10.0.22631', 'SP0', 'Multiprocessor Free')
Interface language: None
EXE path: C:\Program Files\Calibre2\calibre.exe
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (2, 13, 0) && DeDRM (10, 0, 9) && Package KFX (from KFX Input) (2, 13, 0) && Embed Comic Metadata (1, 6, 6) && EpubMerge (2, 18, 0) && Goodreads (1, 8, 2) && Goodreads Sync (1, 16, 8) && KFX metadata reader (from KFX Input) (2, 13, 0) && From KFX (2, 13, 0) && KFX Input (2, 13, 0) && Set KFX metadata (from KFX Output) (2, 8, 0) && KFX Output (2, 8, 0) && Manage Series (1, 5, 2)
Traceback (most recent call last):
File "calibre\gui2\convert\single.py", line 304, in accept
File "calibre\gui2\convert\metadata.py", line 256, in pre_commit_check
File "calibre\db\cache.py", line 85, in call_func_with_lock
File "calibre\db\cache.py", line 1634, in set_field
File "calibre\db\write.py", line 572, in set_books
File "calibre\db\write.py", line 491, in many_many
File "calibre\db\backend.py", line 1159, in executemany
File "C:\t\t\apsw-42s16m2x\src\cursor.c", line 240, in resetcursor
apsw.ConstraintError: ConstraintError: Foreign key violation: tags is still referenced

[laxarus]

uhm. Nevermind, I found out the issue which was totally unrelated to KFX. For future reference to anyone who might come across this, the readarr is sharing the library with calibre and for some reason, it is preventing the conversion. Stopped readarr docker container and tried converting.

[jhowell]

Version 2.14.0 - 19 Jun 2024

Fix incorrect handling of local_delta_fragments in Scribe notebooks. (Fixes "Book appears to have KFX containers from multiple books. (duplicate fragments)")

[tomsem]

So I finally worked out how to install the necessary components to be able to decode and import KFX using Kindle Classic on macOS. I have been just doing AZW3 download, but that doesn't work for Print Replica, and I have some number of these in my Kindle Library. So thank you for your work! (Now I just need to find the rest of them so I can add those to calibre as well)

So I have a functional KFX, and have converted to PDF and fixed layout ePub using From KFX.

A couple of questions:

The PDF has no table of contents (PDF outline items or Bookmarks). I have to reproduce them manually. 'It would be nice' if KFX Input could add these. It looks like you are using PyPDF, which I believe can do this.

The fixed layout ePub has a table of contents, but the text is not searchable or selectable.

I thought I had succeeded awhile back in generating a toy fixed layout ePub with InDesign that had these properties, but just tried it again, and the text is searchable, but not selectable.

The only fixed layout ePubs I have laying around are those generated for comics/manga, so I don't know what I should be expecting.

[sashsz]

Amazon published some new update to Kindle and the new books seem to not be able to get unprotected. I used to use LD Player for this purpose. Now it doesn't work.
What I noticed is that the new book files are .kfx. There is some new file called amzn1.drm-voucher.v1. I think there might be the key to unlock the kfx archive file with the book contents.

[jhowell]

Originally Posted by tomsem

The PDF has no table of contents (PDF outline items or Bookmarks). I have to reproduce them manually. 'It would be nice' if KFX Input could add these. It looks like you are using PyPDF, which I believe can do this.

It should be possible to create a PDF outline from the book's table of contents. I will add that feature unless I run into a problem in implementing it.

Originally Posted by tomsem

The fixed layout ePub has a table of contents, but the text is not searchable or selectable.

EPUB fixed layout created by this plugin from Print Replica retains the structure of Amazon's format by creating visible page images along with a hidden layer of unformatted text. That differs from the way fixed layout books are usually designed and prevents them from working well in most reader software. I recommend converting Print Replica to PDF instead.

[jhowell]

Originally Posted by sashsz

Amazon published some new update to Kindle and the new books seem to not be able to get unprotected. I used to use LD Player for this purpose. Now it doesn't work.
What I noticed is that the new book files are .kfx. There is some new file called amzn1.drm-voucher.v1. I think there might be the key to unlock the kfx archive file with the book contents.

It appears that you are using a recent version of the Kindle for Android app in your emulator to download your books. Doing that will produce books in KFX format but you will be unable to obtain the key needed to unlock them.

See the thread DRM workaround: Kindle for Android for more information on that technique. If you do it properly with an old enough version of Kindle for Android you can avoid KFX format and remove the DRM.

[tomsem]

Originally Posted by jhowell

It should be possible to create a PDF outline from the book's table of contents. I will add that feature unless I run into a problem in implementing it.



EPUB fixed layout created by this plugin from Print Replica retains the structure of Amazon's format by creating visible page images along with a hidden layer of unformatted text. That differs from the way fixed layout books are usually designed and prevents them from working well in most reader software. I recommend converting Print Replica to PDF instead.

Yes, I'll stick with PDF, thanks.

[sashsz]

Originally Posted by jhowell

It appears that you are using a recent version of the Kindle for Android app in your emulator to download your books. Doing that will produce books in KFX format but you will be unable to obtain the key needed to unlock them.

See the thread DRM workaround: Kindle for Android for more information on that technique. If you do it properly with an old enough version of Kindle for Android you can avoid KFX format and remove the DRM.

Hello,
I am using an old version as per the post that you mentioned, as I had read about the initial solution and implemented it sometime ago. Amazon has done something recently and the new books require the latest Kindle version where the book format is KFX. So, I updated to the latest in order to check out the new book that I purchased. There is no way to download the new books unless upgrade to the latest Kindle version!
So, old books are readable and still in azw3,mobi, etc.

[jhowell]

Originally Posted by sashsz

Hello,
I am using an old version as per the post that you mentioned, as I had read about the initial solution and implemented it sometime ago. Amazon has done something recently and the new books require the latest Kindle version where the book format is KFX. So, I updated to the latest in order to check out the new book that I purchased. There is no way to download the new books unless upgrade to the latest Kindle version!
So, old books are readable and still in azw3,mobi, etc.

Amazon is known for occasionally taking steps to block DRM removal methods. If using an old version of the Kindle for Android app is no longer working for newer books then I suggest trying one of the other methods mentioned in the thread Dealing with Kindle ebooks in calibre.

[tomsem]

Originally Posted by tomsem

Yes, I'll stick with PDF, thanks.

One of the Print Replica books in my library was free, and available from Play Books also, available as both fixed-layout ePub and PDF. The fixed-layout ePub consists of only a series of images, most likely something Google generated.

There is a very lengthy document over on W3.org, recently posted or updated:

EPUB Fixed Layout Accessibility

I just wish they would include an example ePub illustrating the 'best practices'.

I just tried opening the ePub generated from print replica by KFX Input with Thorium Reader, it is searchable and text-to-speech works (I had tried Apple Books before). So that seems to be 'as expected', and maybe as much as is possible (i.e. 'selectability' isn't supported).

[tomsem]

As a Scribe owner, the Print Replica KFX the plugin produces is of value without any conversion, i.e. for side loading.

It behaves just as if I'd been able to download it directly, I can write on it, it downloads a cover thumbnail, it syncs with the Kindle apps, and I can export Notes with handwritten annotations.

Some day perhaps the stars will align and publishers will enable Scribe for downloading their print replica books directly, but I'm happy this workaround exists.

[jhowell]

Originally Posted by tomsem

Some day perhaps the stars will align and publishers will enable Scribe for downloading their print replica books directly, but I'm happy this workaround exists.

eBooks you can write on are print replica specifically delivered only to Scribe devices. Other print replica books are most disallowed on Kindle devices including Scribes. I haven't been able to determine why some are allowed and other not, but I doubt that Amazon is giving the choice to publishers.

[tomsem]

Originally Posted by jhowell

eBooks you can write on are print replica specifically delivered only to Scribe devices. Other print replica books are most disallowed on Kindle devices including Scribes. I haven't been able to determine why some are allowed and other not, but I doubt that Amazon is giving the choice to publishers.

Yes, write on books are a sort of scam, when you can easily find a PDF that will provide the same experience (word puzzles, sudoku, journaling guides, etc.) for free or with better value (Dell sells many puzzle books). They are only available for Scribe; nothing else can write on them.

KDP information at least says you can choose supported devices for Print Replica. I assume publishers in general have the same flexibility.

The only thing I think that might bother publishers is the way Notes are exported: the PDF contains large, readable thumbnails for as many pages as you have added handwritten annotations to, maybe they don't like how easy it is to share that via email? But Amazon could add a flag to disallow handwritten annotations if that were the issue (and for 'symmetry' with reflowable books, I would like to see PR support handwritten sticky notes as well as text notes).

Given that PR is used for many textbooks, sheet music, etc. it would make Scribe a more attractive device for students (or put another way, the restriction makes it less attractive for that purpose, effectively undermining any effort to promote it in education market).

There are some PR books that are available for both the Kindle apps and for Scribe (alone of all Kindles):

Kanji From Zero! 1

[Birger]

HI,

My first post here.
I got the following error when converting a book from KFX to ePUB:

Spoiler:

calibre, version 7.12.0
ERROR: KFX conversion failed:
Cannot convert The Design Thinking Toolbox
Exception('This book has a layout that is incompatible with calibre conversion. For best results use either the From KFX user interface plugin or the KFX Input plugin CLI for conversion. See the KFX Input plugin documentation for more information.')

I know the KFX plugin is installed and active as well as DeDRM as I have converted other books.

I cannot see anything suspicious in the debug log:
ApplicationPaletteChange event ignored
Using calibre Qt style: True
ApplicationPaletteChange event received
calibre Debug log
calibre 7.12 embedded-python: True
Windows-10-10.0.22631-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22631')
Python 3.11.5
Windows: ('10', '10.0.22631', 'SP0', 'Multiprocessor Free')
Interface language: None
EXE path: C:\Program Files\Calibre2\calibre-debug.exe
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (2, 14, 0) && DeDRM (10, 0, 9) && Package KFX (from KFX Input) (2, 14, 0) && KFX metadata reader (from KFX Input) (2, 14, 0) && From KFX (2, 14, 0) && KFX Input (2, 14, 0)
calibre 7.12 embedded-python: True
Windows-10-10.0.22631-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22631')
Python 3.11.5
Windows: ('10', '10.0.22631', 'SP0', 'Multiprocessor Free')
Interface language: None
EXE path: C:\Program Files\Calibre2\calibre-debug.exe
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (2, 14, 0) && DeDRM (10, 0, 9) && Package KFX (from KFX Input) (2, 14, 0) && KFX metadata reader (from KFX Input) (2, 14, 0) && From KFX (2, 14, 0) && KFX Input (2, 14, 0)
QPA platform: windows
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 109.00922045264039 x 108.85714285714285
[0.00] Starting up...
[0.00] Showing splash screen...
[0.16] splash screen shown
[0.16] Initializing db...
[0.22] db initialized
[0.22] Constructing main UI...
[1.16] GUI main window shown
[1.23] main UI initialized...
[1.23] Hiding splash screen

Setting that it continues on error made it complete the conversion, but the ePUB file is much larger than the Kindle file and in the file some pages are both images of the page as well as text from the page. The book has lots of figures and it seems like the conversion fails on those.

Is there anything I can do to find out what goes wrong?