Handling KFX files in Calibre

[HarryT]

Originally Posted by pdurrant

Yes, once someone writes a converter. But bear in mind that KFX is 'half-compiled', so it'll be a rather strange ePub one gets out of it.

One difficulty is that Calibre is designed to handle books that are in a single file, but a KFX book consists of the book file itself, and then an auxiliary folder containing images, etc. Any attempt to store KFX files in Calibre would presumably necessitate putting the various components of the book in a ZIP file (or something similar), and the Kindle device drivers then modifying to unpack them again when the book was saved to disk or sent to the device.

[jhowell]

Originally Posted by HarryT

One difficulty is that Calibre is designed to handle books that are in a single file, but a KFX book consists of the book file itself, and then an auxiliary folder containing images, etc. Any attempt to store KFX files in Calibre would presumably necessitate putting the various components of the book in a ZIP file (or something similar), and the Kindle device drivers then modifying to unpack them again when the book was saved to disk or sent to the device.

KFX files are composed of a set of records, much like Amazon's MOBI or KF8 formats. While Amazon delivers books with those records split across several files, it is possible to combine them into a single monolithic KFX file that can be read in Amazon's various apps and devices. This would be more suitable for use within calibre.

I think that the best way to handle this would be to have a calibre FileTypePlugin that, when given a KFX file to import, locates the rest of the files making up that book (the file organization is different for various apps/devices), calls the DRM Tools plugin to remove encryption from those that have it, and combines them into a single KFX file.

A separate conversion input plugin could then be used to convert from single-file KFX to EPUB.

[kovidgoyal]

@jhowell: Not quite. No part of calibre can depend on DRM removal plugins. So the way to go would be to have the device driver/file type plugin assemble the split KFX into a monolothic (.kfx extension) but still DRMed file, or if that is not possible, into a ZIP file containing all the components (.kfx-zip extension).

The DeDRM plugin would then need to remove the DRM from both .kfx and .kfx-zip and return a .kfx without DRM, just as it does for EPUB or MOBI files.

The conversion input plugin would then handle DRM free .kfx files.

[jhowell]

Originally Posted by kovidgoyal

@jhowell: Not quite. No part of calibre can depend on DRM removal plugins. So the way to go would be to have the device driver/file type plugin assemble the split KFX into a monolothic (.kfx extension) but still DRMed file, or if that is not possible, into a ZIP file containing all the components (.kfx-zip extension).

Oops. I didn't consider the importance of keeping DRM removal completely separate in my proposed solution. I think your second alternate, zipping the various files, is the most practical.

----

I realize now that I do not understand how calibre handles file import using File Type Plugins.

What happens when multiple File Type Plugins are registered to handle the same format?

What happens if a plugin changes the format of an imported book to one that another plugin can handle?

(Perhaps these questions are off-topic for this thread.)

[HarryT]

They would perhaps be more appropriate questions for the Calibre forum, I think.

[kovidgoyal]

@jhowell: Nothing fancy. All plugins for the initial file type are called in arbitrary sequence. If the filetype is changed by plugin A then either plugin B will raise and error or handle the file, in either case it is processed.

The DeDRM plugin will need to be able to handle all filetypes (i.e. .kfx (monolithic), .kfx (split) and .kfx-zip). The device driver can have some non-plugin code to assemble split kfx into kfx-zip. So the device driver will always yield either monolothic kfx or kfx-zip. The DEDRM plugin will always yield monolithic kfx.

@HarryT: feel free to move these posts into their own thread, I'm not a moderator for this subforum.

[HarryT]

I've moved the Calibre-related posts to the Calibre forum.

[Katsunami]

Originally Posted by HarryT

One difficulty is that Calibre is designed to handle books that are in a single file, but a KFX book consists of the book file itself, and then an auxiliary folder containing images, etc. Any attempt to store KFX files in Calibre would presumably necessitate putting the various components of the book in a ZIP file (or something similar), and the Kindle device drivers then modifying to unpack them again when the book was saved to disk or sent to the device.

Not to mention that, if you want an EPUB, all the files would need to be unpacked, the EPUB created from scratch (directory structure, etc), all links would need to be changed, and if everything is done, the file would need to be repacked. While repacking an EPUB file is probably already in the Calibre API, combining all the KFX files into an EPUB-valid structure and rewriting all the links isn't. A completely new plugin would need to be written, debugged, and supported.

[ericvancauwenber]

Originally Posted by jhowell

KFX files are composed of a set of records, much like Amazon's MOBI or KF8 formats. While Amazon delivers books with those records split across several files, it is possible to combine them into a single monolithic KFX file that can be read in Amazon's various apps and devices. This would be more suitable for use within calibre.

I think that the best way to handle this would be to have a calibre FileTypePlugin that, when given a KFX file to import, locates the rest of the files making up that book (the file organization is different for various apps/devices), calls the DRM Tools plugin to remove encryption from those that have it, and combines them into a single KFX file.

A separate conversion input plugin could then be used to convert from single-file KFX to EPUB.

HELLO I am new and have an problem with converting from KFX to EPUB with the program CALIBRE It gives always the error : cannot be convert , the file is KFX And I have placed the "kfx plugin " in the Calbre program WHAT IS WRONG ??? THANKS

[DNSB]

Originally Posted by ericvancauwenber

HELLO I am new and have an problem with converting from KFX to EPUB with the program CALIBRE It gives always the error : cannot be convert , the file is KFX And I have placed the "kfx plugin " in the Calbre program WHAT IS WRONG ??? THANKS

Did you install the KFX Input or KFX Output plugins? They are two separate plugins. Do you have deDRM setup to handle KFX files?

[jhowell]

Originally Posted by ericvancauwenber

HELLO I am new and have an problem with converting from KFX to EPUB with the program CALIBRE It gives always the error : cannot be convert...

When importing a book from Kindle for PC/Mac the original file name will have an ".azw" extension. If the import process fails it can result in the book showing AZW format within your calibre library. calibre will treat that as being in a different Kindle e-book format than KFX, leading the error "Failed to convert: (name) This is an Amazon KFX book. It cannot be processed...." when attempting to convert it to another format.

Some things to check:

Make sure that you have both the KFX Input plugin and the DeDRM plugin installed and that you restarted calibre after installation.

When you import the main book file it must be in its original location in a subfolder of the "My Kindle Content" folder.

If you can't resolve the problem you will need to create and post a log of the import process in order to determine what is going wrong: Click the Preferences drop-down menu and choose "Restart in debug mode". Once calibre has re-started, import the problem ebook. Now close calibre. A log will appear that you can copy and paste.

[lalitsheoran]

Originally Posted by jhowell

When importing a book from Kindle for PC/Mac the original file name will have an ".azw" extension. If the import process fails it can result in the book showing AZW format within your calibre library. calibre will treat that as being in a different Kindle e-book format than KFX, leading the error "Failed to convert: (name) This is an Amazon KFX book. It cannot be processed...." when attempting to convert it to another format.

Some things to check:

Make sure that you have both the KFX Input plugin and the DeDRM plugin installed and that you restarted calibre after installation.

When you import the main book file it must be in its original location in a subfolder of the "My Kindle Content" folder.

If you can't resolve the problem you will need to create and post a log of the import process in order to determine what is going wrong: Click the Preferences drop-down menu and choose "Restart in debug mode". Once calibre has re-started, import the problem ebook. Now close calibre. A log will appear that you can copy and paste.

I tried importing 3 different kfx ebooks but they all got imported as kfx-zip , i can see their cover and other meta info in caliber but cant open or convert those. Here is the log -

Spoiler:

calibre Debug log
calibre 3.4 embedded-python: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.12+
Windows: ('7', '6.1.7601', 'SP1', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP File Type (from KFX Input) (1, 4, 0) && DeDRM (6, 6, 0) && Package KFX File Type (from KFX Input) (1, 4, 0) && KindleUnpack - The Plugin (0, 81, 5) && KFX metadata reader (from KFX Input) (1, 4, 0) && KFX Input (1, 4, 0)
calibre 3.4 embedded-python: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.12+
Windows: ('7', '6.1.7601', 'SP1', u'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP File Type (from KFX Input) (1, 4, 0) && DeDRM (6, 6, 0) && Package KFX File Type (from KFX Input) (1, 4, 0) && KindleUnpack - The Plugin (0, 81, 5) && KFX metadata reader (from KFX Input) (1, 4, 0) && KFX Input (1, 4, 0)
Turning on automatic hidpi scaling
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 72.0487534626 x 71.9822878229
Using calibre Qt style: True
Starting up...
Started up in 61.06 seconds with 133 books
stdout+stderr from file dialog helper: ['', '']
piped data from file dialog helper: ['td\xdee\xe6\x02jZE\xcel\xe4\xd0\xaf\x9ej\xdft2\xf 5\x08\x80\x85;\xfee\xd7\x18\xa6\xc3h\xb7', 'C:\\Users\\LaLiT\\Documents\\My Kindle Content\\B01MFF4WAV_EBOK\\B01MFF4WAV_EBOK.azw']
Gather KFX-ZIP File Type (from KFX Input) 1.4.0: Importing C:\Users\LaLiT\Documents\My Kindle Content\B01MFF4WAV_EBOK\B01MFF4WAV_EBOK.azw
Gather KFX-ZIP File Type (from KFX Input): Gathered 6 files as C:\Users\LaLiT\AppData\Local\Temp\calibre_cqevro\j 3xhgd.kfx-zip
DeDRM v6.6.0: Trying to decrypt j3xhgd.kfx-zip
Using Library AlfCrypto Python
Decrypting KFX-ZIP ebook: j3xhgd
Got DSN key from database default_key
Found 1 keys to try after 0.1 seconds
Decrypting KFX DRM voucher: amzn1.drm-voucher.v1.b63c2321-5bde-4dec-9cf1-5070e5f5af6d.voucher
KFX DRM voucher successfully decrypted
DeDRM v6.6.0: Failed to decrypt with error: This book is licensed as KindleUnlimited. These tools are intended for use on purchased books.
DeDRM v6.6.0: Looking for new default Kindle Key after 0.2 seconds
searching for kinfoFiles in C:\Users\LaLiT\AppData\Local
Found K4PC 1.9+ kinf2011 file: C:\Users\LaLiT\AppData\Local\Amazon\Kindle\storage \.kinf2011
Decrypted key file using IDString '115143377' and UserName '4c614c6954'
DeDRM v6.6.0: Found 1 new key
Decrypting KFX-ZIP ebook: j3xhgd
Got DSN key from database default_key_1
Found 1 keys to try after 0.4 seconds
Decrypting KFX DRM voucher: amzn1.drm-voucher.v1.b63c2321-5bde-4dec-9cf1-5070e5f5af6d.voucher
KFX DRM voucher successfully decrypted
DeDRM v6.6.0: Ultimately failed to decrypt after 0.4 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/...master/FAQs.md
Running file type plugin DeDRM failed with traceback:
Traceback (most recent call last):
File "site-packages\calibre\customize\ui.py", line 171, in _run_filetype_plugins
File "calibre_plugins.dedrm.__init__", line 620, in run
File "calibre_plugins.dedrm.__init__", line 570, in KindleMobiDecrypt
DeDRMError: DeDRM v6.6.0: Ultimately failed to decrypt after 0.4 seconds. Read the FAQs at Harper's repository: https://github.com/apprenticeharper/...master/FAQs.md
Package KFX File Type (from KFX Input) 1.4.0: Packaging C:\Users\LaLiT\AppData\Local\Temp\calibre_cqevro\j 3xhgd.kfx-zip
Processing container: B01MFF4WAV_EBOK.azw
Running file type plugin Package KFX File Type (from KFX Input) failed with traceback:
Traceback (most recent call last):
File "site-packages\calibre\customize\ui.py", line 171, in _run_filetype_plugins
File "calibre_plugins.kfx_input.package_filetype", line 22, in run
File "calibre_plugins.kfx_input.package_filetype", line 35, in run_kfx
File "calibre_plugins.kfx_input.kfxlib.yj_book", line 52, in convert_to_single_kfx
File "calibre_plugins.kfx_input.kfxlib.yj_book", line 118, in decode_book
File "calibre_plugins.kfx_input.kfxlib.yj_book", line 205, in get_container
KFXDRMError: Book container B01MFF4WAV_EBOK.azw has DRM and cannot be converted
KFX metadata reader (from KFX Input) activated for C:\Users\LaLiT\AppData\Local\Temp\calibre_cqevro\o mohp7_add_books\0\B01MFF4WAV_EBOK.kfx-zip
Added Nithalle Ki Diary (Hindi) to db in: 1.6
Added 1 books in 4.1 seconds

[latepaul]

The answer is in the error message:

Originally Posted by lalitsheoran

DeDRM v6.6.0: Failed to decrypt with error: This book is licensed as KindleUnlimited. These tools are intended for use on purchased books.

[Holden100]

Originally Posted by latepaul

The answer is in the error message:

Ah, so there is no way for Kindle Unlimited other than Epubor Ultimate?

[darryl]

Originally Posted by Holden100

Ah, so there is no way for Kindle Unlimited other than Epubor Ultimate?

Epubor uses the open source DeDrm tools without complying with the licence or in fact even acknowledging it. The DeDrm tools do not assist you in removing the DRM from books you have no right to keep permanently, such as books borrowed from the library or from Kindle Unlimited. It would be perfectly in keeping with the ethics of those behind Epubor to change the code to allow you to do this. Do they advertise that it works on KU and/or borrowed books? I couldn't see it on their site. However, their faq says;

Handle rented Kindle books
An important premise for removing DRM is you should be the owner of the book, in other words, the books must be purchased by you.

Therefore, decrypting rented books is not supported by Epubor software, sorry for the inconvenience.

It would seem that even Epubor will not do what you want. Might I suggest that if you want to keep a borrowed book to read that you seriously consider buying it.