[Conversion Plugin] KFX Input

[Jaws]

Originally Posted by jhowell

However if you are using From KFX to generate the EPUB and doing no further conversion after that then the set of HTML files in the EPUB produced reflects the actual internal structure of the book as provided by Amazon.

The only conversion is (1) import to Calibre using KFXInput; (2) use Calibre's Convert Books | Individually to change the format from KFX to Epub, with no other conversions (etc.) and with the heuristics off and otherwise default preferences. At that point, Edit Book discloses the file splitting, as does exporting to other readers without using Edit Book (because of the extreme "new page" problem discussed in the original post).

Originally Posted by jhowell

If you can provide the title and author or ASIN of some books that exhibits this problem I may be able to tell from the free samples whether the file splitting came that way from the publisher or was done later.

Here's an example (that should be easy for US people to get):
B0BDCR5LQC
It certainly wasn't the first one I noticed this problem with; it's much, much worse (and makes actually reading the Epub much harder) with a lot of 1980s-2000s chemistry and biology texts, where a single-line formula that has been converted to a graphic gets its own page. I'd suspect something with LaTEX except that nobody uses LaTEX for cookbooks.

[jhowell]

Originally Posted by Jaws

Here's an example (that should be easy for US people to get):
B0BDCR5LQC
It certainly wasn't the first one I noticed this problem with; it's much, much worse (and makes actually reading the Epub much harder) with a lot of 1980s-2000s chemistry and biology texts, where a single-line formula that has been converted to a graphic gets its own page. I'd suspect something with LaTEX except that nobody uses LaTEX for cookbooks.

I took a look at the free sample of that book and found that it makes frequent use of page-break-before and page-break-after to force images to each appear on their own page. This was apparently done to mimic the formatting of the print edition of that book.

Conversion to KFX format usually replaces page breaks with actual splitting of the content into separate files, and that is the case with this book. While the sample in AZW3 format contains 15 HTML files the equivalent KFX version has 27.

Of the Kindle formats KF8 (AZW3) most closely reflects what the publisher provided to Amazon. KFX is highly processed and when converted to EPUB can differ significantly in the details of HTML formatting.

If retaining the original formatting is important to you I suggest looking into ways of avoiding KFX, such as using Download & Transfer via USB from Amazon's website.

[Bresaola0]

Hello,

i looked for the solution in this post, but i couldn't find it, and i also wanted to apologise in advance because i'm not very experienced, in any case i'm attaching the error message, does anyone know how to solve it? i've tried converting the file from calibre, directly from KFX imput enabling all the ticks but it doesn't work

[jhowell]

Originally Posted by Bresaola0

i looked for the solution in this post, but i couldn't find it, and i also wanted to apologise in advance because i'm not very experienced, in any case i'm attaching the error message, does anyone know how to solve it? i've tried converting the file from calibre, directly from KFX imput enabling all the ticks but it doesn't work

For best results see the section "From KFX" toolbar action in the first post in this thread for instructions on how to bypass calibre conversion. Or enable the 'Allow conversion to complete even if errors are detected' conversion option in the KFX Input tab to proceed with calibre conversion. However the results will not be as good if you do it that way.

[yoshi]

We can convert KFX to EPUB using "From KFX."
Are there any disadvantages compared to using Calibre's "Convert books" besides the fact that conversions must be done one book at a time?

[jhowell]

Originally Posted by yoshi

We can convert KFX to EPUB using "From KFX."
Are there any disadvantages compared to using Calibre's "Convert books" besides the fact that conversions must be done one book at a time?

The main disadvantage is that because it bypasses the built in conversion mechanism of calibre it does not allow any of the optional handling that calibre can perform, such as changing the layout or updating metadata.

The advantage is that the internal formatting of the resulting EPUB will better match the book as published. That can make a significant difference for books that use fixed-layout formatting since that is not supported by calibre and using calibre's conversion on such a book will typically result in an unreadable mess.

[PikeUK]

I have two books in my Kindle library (from Amazon UK) that give the following error after downloading in Kindle for Windows v2.4.70904, importing into Calibre v7.26 with latest DeDRM and KFX Input 2.19.0, then trying to convert to EPUB:

Exception('Unexpected Ion symbols used: $398')

They are:
Perdido Street Station by China Miéville (Pan Macmillan, Aug 2008)

Spoiler:

Convert book 755 of 1048 (Perdido Street Station)
DeDRM v10.0.9: Trying to decrypt zywvs7ha.kfx
Unknown booktype kfx. Passing back to calibre unchanged
Conversion options changed from defaults:
cover: 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\z6qsjb2a.jpeg'
output_profile: 'generic_eink'
read_metadata_from_opf: 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\2402o4cb.opf'
verbose: 2
Resolved conversion options
calibre version: 7.26.0
{'add_alt_text_to_img': False,
'allow_conversion_with_errors': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0.0,
'book_producer': None,
'change_justification': 'original',
'chapter': "//*[((name()='h1' or name()='h2') and re:test(., "
"'\\s*((chapter|book|section|part)\\s+)|((prolog|p rologue|epilogue)(\\s+|$))', "
"'i')) or @class = 'chapter']",
'chapter_mark': 'pagebreak',
'comments': None,
'cover': 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\z6qsjb2a.jpeg',
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': '',
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000024623AA7FD0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x0000024623AB3950>,
'page_breaks_before': '/',
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\2402o4cb.opf',
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': '[]',
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': None,
'sr1_search': None,
'sr2_replace': None,
'sr2_search': None,
'sr3_replace': None,
'sr3_search': None,
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': '[]',
'transform_html_rules': '[]',
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
DeDRM v10.0.9: Trying to decrypt zywvs7ha.kfx
Unknown booktype kfx. Passing back to calibre unchanged
InputFormatPlugin: KFX Input running
on C:\Users\PikeUK\AppData\Local\Temp\calibre_covllw0 2\zywvs7ha.kfx
Software versions: KFX Input 2.19.0, calibre 7.26, Windows-10-10.0.26100-SP0
KFX Input plugin help is available at https://www.mobileread.com/forums/sh...d.php?t=291290
Converting C:\Users\PikeUK\AppData\Local\Temp\calibre_covllw0 2\zywvs7ha.kfx
Processing container: C:\Users\PikeUK\AppData\Local\Temp\calibre_covllw0 2\zywvs7ha.kfx
Features: kfxgen.textBlock-1, reflow-style-1
Metadata: ASIN=B003GK21A8, asset_id=CR!DRTR58ZY5S1JK471G9ZN35W8N0G2, author="Miéville, China", author="China Miéville", cde_content_type=EBOK, content_id=B003GK21A8, cover_image=678x1024, creator_version=2.15.0, file_creator=YJConversionTools, is_sample=False, issue_date=2008-08-28, kfxgen="6.28.1.0/PackageVersion:YJReaderSDK-1.1.67.4 Month-Day:07-14", language=en, max_id=609, min_kindle_version=5.6.5, override_kindle_font=False, pages=723, publisher="Pan Macmillan", publisher="Pan Books", reading_orders=1, selection=enabled, title="Perdido Street Station"
ERROR: Unexpected Ion symbols used: $398
Converting book to EPUB 2
Book symbol format is original
Traceback (most recent call last):
File "calibre_plugins.kfx_input.__init__", line 118, in convert
Exception: Unexpected Ion symbols used: $398

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 38, in gui_convert_override
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1128, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre_plugins.kfx_input.__init__", line 124, in convert
calibre.ebooks.conversion.ConversionUserFeedBack: {"msg": "<b>Cannot convert Perdido Street Station</b><br><br>Exception('Unexpected Ion symbols used: $398')", "level": "error", "det_msg": "", "title": "KFX conversion failed"}

The Rithmatist by Brandon Sanderson (Pan Macmillan, May 2014)

Spoiler:

Convert book 814 of 1048 (The Rithmatist)
DeDRM v10.0.9: Trying to decrypt 5o2nxnbf.kfx
Unknown booktype kfx. Passing back to calibre unchanged
Conversion options changed from defaults:
read_metadata_from_opf: 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\a2q4l0me.opf'
verbose: 2
cover: 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\pldea60j.jpeg'
output_profile: 'generic_eink'
Resolved conversion options
calibre version: 7.26.0
{'add_alt_text_to_img': False,
'allow_conversion_with_errors': False,
'asciiize': False,
'author_sort': None,
'authors': None,
'base_font_size': 0.0,
'book_producer': None,
'change_justification': 'original',
'chapter': "//*[((name()='h1' or name()='h2') and re:test(., "
"'\\s*((chapter|book|section|part)\\s+)|((prolog|p rologue|epilogue)(\\s+|$))', "
"'i')) or @class = 'chapter']",
'chapter_mark': 'pagebreak',
'comments': None,
'cover': 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\pldea60j.jpeg',
'debug_pipeline': None,
'dehyphenate': True,
'delete_blank_paragraphs': True,
'disable_font_rescaling': False,
'dont_split_on_page_breaks': False,
'duplicate_links_in_toc': False,
'embed_all_fonts': False,
'embed_font_family': None,
'enable_heuristics': False,
'epub_flatten': False,
'epub_inline_toc': False,
'epub_max_image_size': 'none',
'epub_toc_at_end': False,
'epub_version': '2',
'expand_css': False,
'extra_css': None,
'extract_to': None,
'filter_css': '',
'fix_indents': True,
'flow_size': 260,
'font_size_mapping': None,
'format_scene_breaks': True,
'html_unwrap_factor': 0.4,
'input_encoding': None,
'input_profile': <calibre.customize.profiles.InputProfile object at 0x0000029D0BE57FD0>,
'insert_blank_line': False,
'insert_blank_line_size': 0.5,
'insert_metadata': False,
'isbn': None,
'italicize_common_cases': True,
'keep_ligatures': False,
'language': None,
'level1_toc': None,
'level2_toc': None,
'level3_toc': None,
'line_height': 0.0,
'linearize_tables': False,
'margin_bottom': 5.0,
'margin_left': 5.0,
'margin_right': 5.0,
'margin_top': 5.0,
'markup_chapter_headings': True,
'max_toc_links': 50,
'minimum_line_height': 120.0,
'no_chapters_in_toc': False,
'no_default_epub_cover': False,
'no_inline_navbars': False,
'no_svg_cover': False,
'output_profile': <calibre.customize.profiles.GenericEink object at 0x0000029D0BE63910>,
'page_breaks_before': '/',
'prefer_metadata_cover': False,
'preserve_cover_aspect_ratio': False,
'pretty_print': True,
'pubdate': None,
'publisher': None,
'rating': None,
'read_metadata_from_opf': 'C:\\Users\\PikeUK\\AppData\\Local\\Temp\\calibre_ covllw02\\a2q4l0me.opf',
'remove_fake_margins': True,
'remove_first_image': False,
'remove_paragraph_spacing': False,
'remove_paragraph_spacing_indent_size': 1.5,
'renumber_headings': True,
'replace_scene_breaks': '',
'search_replace': '[]',
'series': None,
'series_index': None,
'smarten_punctuation': False,
'sr1_replace': None,
'sr1_search': None,
'sr2_replace': None,
'sr2_search': None,
'sr3_replace': None,
'sr3_search': None,
'start_reading_at': None,
'subset_embedded_fonts': False,
'tags': None,
'timestamp': None,
'title': None,
'title_sort': None,
'toc_filter': None,
'toc_threshold': 6,
'toc_title': None,
'transform_css_rules': '[]',
'transform_html_rules': '[]',
'unsmarten_punctuation': False,
'unwrap_lines': True,
'use_auto_toc': False,
'verbose': 2}
DeDRM v10.0.9: Trying to decrypt 5o2nxnbf.kfx
Unknown booktype kfx. Passing back to calibre unchanged
InputFormatPlugin: KFX Input running
on C:\Users\PikeUK\AppData\Local\Temp\calibre_covllw0 2\5o2nxnbf.kfx
Software versions: KFX Input 2.19.0, calibre 7.26, Windows-10-10.0.26100-SP0
KFX Input plugin help is available at https://www.mobileread.com/forums/sh...d.php?t=291290
Converting C:\Users\PikeUK\AppData\Local\Temp\calibre_covllw0 2\5o2nxnbf.kfx
Processing container: C:\Users\PikeUK\AppData\Local\Temp\calibre_covllw0 2\5o2nxnbf.kfx
Features: kfxgen.textBlock-1, reflow-style-1
Metadata: ASIN=B00ISW7WVC, asset_id=CR!HY8Y27Y95D2A178G70MPM1GM5CHT, author="Sanderson, Brandon", author="Brandon Sanderson", cde_content_type=EBOK, content_id=B00ISW7WVC, cover_image=675x1024, creator_version=2.15.0, file_creator=YJConversionTools, is_sample=False, issue_date=2014-05-22, kfxgen="3.42.1.0/PackageVersion:YJReaderSDK-1.0.2044.4 Month-Day:10-28", language=en, max_id=609, min_kindle_version=5.6.5, publisher="Pan Macmillan", publisher=Tor, reading_orders=1, selection=enabled, title="The Rithmatist"
ERROR: Unexpected Ion symbols used: $398
Converting book to EPUB 2
Book symbol format is original
Traceback (most recent call last):
File "calibre_plugins.kfx_input.__init__", line 118, in convert
Exception: Unexpected Ion symbols used: $398

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "runpy.py", line 198, in _run_module_as_main
File "runpy.py", line 88, in _run_code
File "site.py", line 83, in <module>
File "site.py", line 78, in main
File "site.py", line 50, in run_entry_point
File "calibre\utils\ipc\worker.py", line 215, in main
File "calibre\gui2\convert\gui_conversion.py", line 38, in gui_convert_override
File "calibre\gui2\convert\gui_conversion.py", line 25, in gui_convert
File "calibre\ebooks\conversion\plumber.py", line 1128, in run
File "calibre\customize\conversion.py", line 242, in __call__
File "calibre_plugins.kfx_input.__init__", line 124, in convert
calibre.ebooks.conversion.ConversionUserFeedBack: {"msg": "<b>Cannot convert The Rithmatist</b><br><br>Exception('Unexpected Ion symbols used: $398')", "level": "error", "det_msg": "", "title": "KFX conversion failed"}

I don't see any obvious problems with either book after reconverting with KFX Input errors ignored.

I'm not sure if the above is useful information but I was surprised to find 0 hits on Google for this error. Either way, thank you very much for this plugin, it's let me back up 100s of ebooks, so I appreciate all the work that has gone into it.

[jhowell]

Originally Posted by PikeUK

I have two books in my Kindle library (from Amazon UK) that give the following error after downloading in Kindle for Windows v2.4.70904, importing into Calibre v7.26 with latest DeDRM and KFX Input 2.19.0, then trying to convert to EPUB:

Exception('Unexpected Ion symbols used: $398')

They are:
Perdido Street Station by China Miéville (Pan Macmillan, Aug 2008)

The Rithmatist by Brandon Sanderson (Pan Macmillan, May 2014)

I don't see any obvious problems with either book after reconverting with KFX Input errors ignored.

I'm not sure if the above is useful information but I was surprised to find 0 hits on Google for this error. Either way, thank you very much for this plugin, it's let me back up 100s of ebooks, so I appreciate all the work that has gone into it.

Thank you for the bug report. Based on the ASINs in the logs I see that those editions are not available in my region (USA). I looked at the US edition of Perdido Street Station (ASIN B000FBFO8C) and it does not produce this error.

Those editions available to you appear to have KFX content that I have not yet encountered. Without an example to experiment with I cannot determine what needs to be done to this plugin to support it. If you PM me a link to a KFX file obtained by importing one of those books into calibre I will look into it further. (I understand if you are not comfortable doing that.)