[Conversion Plugin] KFX Input

[misterr0b0t]

Originally Posted by jhowell

The log shows a failure to remove the DRM from the book. If the publisher has chosen to have DRM applied to that book then you will not be able to convert a copy of it obtained from Kindle for Android.

See the link to the FAQ that was given, especially the section titled "Why don't the tools work with Kindle for Android ebooks?".

As I wrote previously:

So I assume because the DeDRM tools failed to remove the DRM from the book, it has DRM. Does that mean there will never be a way to remove DRM from Kindle for Android e-books? Or is it possible to use an older version of the app that has backup enabled? You also said that "That will only work…if you have managed to extract the DRM keys associated with your Android app installation." Is this something I can do?

[jhowell]

Originally Posted by misterr0b0t

So I assume because the DeDRM tools failed to remove the DRM from the book, it has DRM. Does that mean there will never be a way to remove DRM from Kindle for Android e-books? Or is it possible to use an older version of the app that has backup enabled? You also said that "That will only work…if you have managed to extract the DRM keys associated with your Android app installation." Is this something I can do?

MobileRead is not the right place to ask these questions. Detailed discussion of DRM removal techniques is not allowed here.

Try Apprentice Harper’s GitHub or Apprentice Alf’s blog.

[misterr0b0t]

Originally Posted by jhowell

MobileRead is not the right place to ask these questions. Detailed discussion of DRM removal techniques is not allowed here.

Try Apprentice Harper’s GitHub or Apprentice Alf’s blog.

OK, thanks for the help in diagnosing the problem.

[jhowell]

Version 1.3.0 - 09 Apr 2018

Fix error during conversion of books with referenced fonts that are not present: "'unicode' object has no attribute 'list_symbols'"

Support additional KFX features.

[fireattack]

This plug-in doesn't seem to work well with the "azw+azw.res" format.

It will just import azw (with all the low resolution images), instead of using HD images in .res files.

I tried to package the whole B07C3KZ9Z8_EBOK folder into a zip as well as renamed it to kfx-zip, neither works.

Any solution?

Thanks!

[jhowell]

Originally Posted by fireattack

It will just import azw (with all the low resolution images), instead of using HD images in .res files.

The newer versions of Kindle for PC/Mac use the same set of file extensions regardless of the actual format that each book is in. For books in KFX format this plugin will definitely process .azw.res files, because that is the only place that images are held in that format.

However, many books are still delivered in the older MOBI and KF8 formats. Those formats are handled natively by calibre and this plugin has nothing to do with them.

Those MOBI and KF8 books sometimes have an associated .azw.res file that contains HD images in a similar way to the .azw6 files used on e-ink Kindle devices. Calibre ignores those files and I do not know of any method to combine them with the rest of the book. (Software for dumping the contents of these files can be found here.)

Again, the KFX Input plugin has nothing to do with this issue.

[fireattack]

Originally Posted by jhowell

The newer versions of Kindle for PC/Mac use the same set of file extensions regardless of the actual format that each book is in. For books in KFX format this plugin will definitely process .azw.res files, because that is the only place that images are held in that format.

However, many books are still delivered in the older MOBI and KF8 formats. Those formats are handled natively by calibre and this plugin has nothing to do with them.

Those MOBI and KF8 books sometimes have an associated .azw.res file that contains HD images in a similar way to the .azw6 files used on e-ink Kindle devices. Calibre ignores those files and I do not know of any method to combine them with the rest of the book. (Software for dumping the contents of these files can be found here.)

Again, the KFX Input plugin has nothing to do with this issue.

Thanks! That makes sense. I guess I'm confused by this article a little bit, and mistakenly thought simply azw+zaw.res=kfx.

Thank you for clearing it up, and that py2 script works wonderfully.

One thing I want to add, though, is that I originally tried Kindle 1.1.7 (for avoiding kfx), and I did get a different result after re-downloading the book - same azw without res file at all. This further convinced me it's kfx lol.

[jscholes]

First things first, thanks so much for the plugin! I'm having some trouble when using the calibredb CLI tool to add a KFX .azw book. calibre tries to read the metadata using its built-in support for .azw files, which obviously fails. If I ZIP up the files into a .kfx-zip beforehand and then add that from the CLI, all goes as intended. Here's my command and output:

> calibredb.exe add --duplicates "L:\Dropbox\Documents\My Kindle Content\B00KFEJN3Q_EBOK\B00KFEJN3Q_EBOK.azw"
Traceback (most recent call last):
  File "site-packages\calibre\customize\ui.py", line 417, in get_file_type_metadata
  File "site-packages\calibre\customize\builtins.py", line 284, in get_metadata
  File "site-packages\calibre\ebooks\metadata\mobi.py", line 494, in get_metadata
  File "site-packages\calibre\ebooks\mobi\reader\headers.py", line 284, in __init__
  File "site-packages\calibre\ebooks\mobi\reader\headers.py", line 312, in identity
MobiError: Unknown book type: ???|?"
Gather KFX-ZIP File Type (from KFX Input) 1.3.0: Importing L:\Dropbox\Documents\My Kindle Content\B00KFEJN3Q_EBOK\B00KFEJN3Q_EBOK.azw
KFX Input: Gathered 4 files as <redacted>\calibre_temp\calibre_t6twf1\ll_ch9.kfx-zip
DeDRM v6.6.0: Trying to decrypt ll_ch9.kfx-zip
# <DeDRM stuff>
DeDRM v6.6.0: Finished after 0.3 seconds
Package KFX File Type (from KFX Input) 1.3.0: Packaging <redacted>\calibre_temp\calibre_t6twf1\lcksco.kfx-zip
Processing container: B00KFEJN3Q_EBOK.azw
Processing container: CR!1RM96FWSQH7ZZ8854T5S9EA6ZFHW.azw.md
Processing container: CR!EGDG5953SN2Y9474REC37D1B6EWZ.azw.res
kfxgen version: 7.66.1.0/PackageVersion:YJReaderSDK-1.5.195.0 Month-Day:04-20/668
Features: kfxgen.pidMapWithOffset-1, kfxgen.textBlock-1, reflow-style-1
Metadata: ASIN=B00KFEJN3Q, asset_id=CR!BBH3ZQB3XS1BH6WGPYR8GB3Y7F4P, author="Forsyth, Mark", author="Mark Forsyth", cde_content_type=EBOK, content_id=B00KFEJN3Q, cover_image=600x927, creator_version=2.15.0, description=..., file_creator=YJConversionTools, is_sample=False, issue_date=2013-11-07, language=en, override_kindle_font=False, publisher="Icon Books Ltd", reading_orders=1, selection=enabled, title="The Elements of Eloquence: How to Turn the Perfect English Phrase"
KFX Input: Imported as <redacted>\calibre_temp\calibre_t6twf1\avf5gn.kfx
Added book ids: 1

> calibredb.exe list --for-machine --fields all
[
  {
    "author_sort": "Unknown", 
    "authors": "Unknown", 
    "formats": [
      "<redacted>/calibre_library/Unknown/B00KFEJN3Q EBOK (1)/B00KFEJN3Q EBOK - Unknown.kfx"
    ], 
    "id": 1, 
    "identifiers": {}, 
    "isbn": "", 
    "languages": [], 
    "last_modified": "2018-05-16T18:50:52+00:00", 
    "pubdate": "0101-01-01T00:00:00+00:00", 
    "series_index": 1.0, 
    "size": 2187156, 
    "tags": [], 
    "timestamp": "2018-05-16T18:50:49+00:00", 
    "title": "B00KFEJN3Q EBOK", 
    "uuid": "d98de7ea-b34b-4e7f-9c4f-3ea650fe7ef9"
  }
]

Any idea how I can force calibre to use your metadata reader plugin instead of its built-in one when importing from the command line? If I import the same .azw file via the calibre GUI, the metadata is read fine.

[jhowell]

Originally Posted by jscholes

Any idea how I can force calibre to use your metadata reader plugin instead of its built-in one when importing from the command line?

Update: See next post!

[jhowell]

Originally Posted by jscholes

Any idea how I can force calibre to use your metadata reader plugin instead of its built-in one when importing from the command line?

I looked at the calibre source code and determined that the code for adding books via the command line is quite different from that used in the GUI. The "calibredb add" command uses a hard coded set of formats built into calibre from which it can gather metadata. It may be possible to add support for KFX there, but I don't know the code well enough to attempt it myself.

[jscholes]

Thanks for looking into it, I also reported this as a calibre bug this morning although as you say, the code is quite different so it might be one of those "it's a feature, not a bug" cases.

For now I'll ZIP up my KFX books before adding them via the CLI.

[fireattack]

I'm surprised that CLI tool works with 3rd party plugin, albeit partially.

AFAIK ebook-convert.exe doesn't work with plugin at all (at least not the deDRM one).

[jhowell]

Originally Posted by fireattack

AFAIK ebook-convert.exe doesn't work with plugin at all (at least not the deDRM one).

The KFX Input and KFX Output plugins will work with the ebook-convert and ebook-meta commands.

I believe that the DeDRM plugin is written to only be activated when book formats are imported into calibre. It wouldn't be used for conversion even within the GUI.

-----------------

It looks like the problem with "calibredb add" for books in KFX format will be resolved in the next calibre release.

[jscholes]

Originally Posted by fireattack

AFAIK ebook-convert.exe doesn't work with plugin at all (at least not the deDRM one).

The DeDRM plugin works fine when importing books from the command line using the calibredb tool. It doesn't run during conversion at all, whether from the GUI or using ebook-convert.

Originally Posted by jhowell

It looks like the problem with "calibredb add" for books in KFX format will be resolved in the next calibre release.

Yup, super quick turn-around on that one.

[jhowell]

Version 1.4.0 - 21 May 2018

Fix error when converting KPF files produced by Kindle Create version 1.6: CorruptError: malformed database schema (?)

Do not include extra metadata in OPF that could lead to duplicate cover pages when converting comics.

Support additional KFX features.