KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[KevinH]

Hi,

Originally Posted by DiapDealer

Just a heads up: there are three more places in the mobi_opf script where data gets the unescape->escape treatment in addition to the handleTag and handleMetaPairs methods.

Would it make sense to do something similar (full unicode) in those additional three locations?

As long as that text is destined to be inside xml tags in the opf, then yes, we should probably be doing the same thing there as well. I must have missed them.

I also have one fix for the kindleunpack.py script to move a call out of the parameter list for the main routine into the main body to protect it from being inadvertently invoked when multiprocessing is used on Windows machines.

So once we have those fixes in place we should probably make a new release.

Thanks,

Kevin

[adamselene]

AFAIK, no Kindle devices or apps read fonts from KF6/7, so shouldn't we remove them when stripping a combo file to KF7?

[adamselene]

Actually, there's another problem. A KF7 made by running KindleUnpack on a file produced by KindleGen won't open on my Paperwhite at all. It doesn't even display in the document list, and doesn't get indexed, although a .sdr directory is created for it. Bother.

[tomsem]

It seems that the Personal Documents service will not accept KF8-only files, only hybrid files with both mobi and KF8 (or mobi only). And PDS has a 50MB limit. I have a few that are too big to send and was hoping to chop them in half as it were so I would not need to side-load them.

If mobiunpack could include a 'vestigial' mobi segment with the kf8 output, I have a feeling these would be accepted by PDS and one would still be able to realize the space savings of a kf8 only file. (though these would be unreadable on KDS K4iOS, etc.)

I hope to do some experimentation with media queries and kindlegen to confirm that this would make PDS happy.

[Update: turns out so far that I have been able to just kindlestrip these files to get them under 50MB (they included source file archive). It's more than okay that they remain hybrid when stored in PDS, since I might want to read with Kindle for iOS, which does not support KF8 reflowable yet. So I probably won't bother figuring out how to create a placeholder mobi.]

[KevinH]

Hi,

But almost anything can read the older mobi pieces. So please provide a testcase of a small but valid epub that I pass through the latest kindlegen, then unpack and strip it, that will not show on your Paperwhite, and I will try to track down how to fix it.

Thanks,

KevinH

Originally Posted by adamselene

Actually, there's another problem. A KF7 made by running KindleUnpack on a file produced by KindleGen won't open on my Paperwhite at all. It doesn't even display in the document list, and doesn't get indexed, although a .sdr directory is created for it. Bother.

[adamselene]

Originally Posted by KevinH

But almost anything can read the older mobi pieces. So please provide a testcase of a small but valid epub that I pass through the latest kindlegen, then unpack and strip it, that will not show on your Paperwhite, and I will try to track down how to fix it.

It happens on literally every one, but here's a small test case that also uses embedded fonts (see my previous message).

Something even weirder I noticed, though. If I put the KF8 on the Paperwhite first, let it get indexed and appear in the document list, and then overwrite it with the KF7, it stays in the document list but displays an error when I try to open it.

Addendum: The Kindle 2 has no issue with the KF7 that I can see.

[KevinH]

Hi,
I am travelling right now but will download your testcase as soon as I return. In the meanwhile would you please try this earlier fix to see if it helps:

https://www.mobileread.com/forums/sho...&postcount=502

Just replace the mobi_split.py one with the version from that link.

As for fonts, I can not easily remove their sections because the older mobi html references section numbers in the mobi for images and things that may be intermixed with the fonts. I can modify the code to create sections of 0 size to replace them but I am not sure if any Kindles will have trouble with 0 sized sections. If I could prove that all fonts come after all jpegs/images then, they should be removable.

I can provide a test version of mobi-split so that you can test the version with fonts replaced with 0 sized sections. Hopefully the earlier bug fix will fix your issue with the Paperwhite as well.

Thanks,

KevinH

[adamselene]

Originally Posted by KevinH

Hi,
I am travelling right now but will download your testcase as soon as I return. In the meanwhile would you please try this earlier fix to see if it helps:

https://www.mobileread.com/forums/sho...&postcount=502

Just replace the mobi_split.py one with the version from that link.

That seems to have worked.

[KevinH]

Hi,

Your font testcase is missing most of the fonts that are references in Section001.xhtml. So I am not sure how good a testcase this will turn out to be since kindlegen gives many warnings ...

Warning(prcfile):W14028: Following file does not exist :   ../Fonts/Comic_Sans_MS.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/Courier_New.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/Georgia.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/Times_New_Roman.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/Trebuchet_MS.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/Verdana.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/Andika-R.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/CharisSILR.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/DoulosSILR.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/GenR102.ttf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/AGaramondPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/AJensonPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/ArnoPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/BenguiatStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/BerkeleyStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/BriosoPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/CaeciliaLTStd-Roman.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/ChaparralPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/CheltenhamStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/CronosPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/FrutigerLTStd-Roman.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/GalliardStd-Roman.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/GaramondPremrPro.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/HiroshigeStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/HorleyOldStyleMTStd.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/KeplerStd-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/MendozaRomanStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/MinionPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/MinisterStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/MyriadPro-Regular.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/NewAsterLTStd.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/OptimaLTStd.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/PalatinoLTStd-Roman.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/PlantinStd.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/SlimbachStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/TimesNewRomanMTStd.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/UniversLTStd.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/VeljovicStd-Book.otf
Warning(prcfile):W14028: Following file does not exist :   ../Fonts/WarnockPro-Regular.otf

Either way, I added a simple jpeg image to your testcase and them examined the order of the resulting sections in both parts of the mobi (as built by kindlegen 2.8).

Map of Palm DB Sections
    Dec  - Hex : Description
    ---- - ----  -----------
    0000 - 0000: HEADER 6
    0001 - 0001: Text Record 0
    0002 - 0002: NCX Index 0
    0003 - 0003: NCX Index 1
    0004 - 0004: NCX Index CNX
    0005 - 0005: FONT
    0006 - 0006: FONT
    0007 - 0007: FONT
    0008 - 0008: FONT
    0009 - 0009: FONT
    0010 - 000a: FONT
    0011 - 000b: FONT
    0012 - 000c: Image jpeg
    0013 - 000d: RESC
    0014 - 000e: FLIS
    0015 - 000f: FCIS
    0016 - 0010: Source Archive 0
    0017 - 0011: Source Archive 1
    0018 - 0012: BOUNDARY
    0019 - 0013: HEADER 8
    0020 - 0014: Text Record 0
    0021 - 0015: Text Record 1
    0022 - 0016: Text Record 2
    0023 - 0017: 0000
    0024 - 0018: Fragment Index 0
    0025 - 0019: Fragment Index 1
    0026 - 001a: Fragment Index CNX
    0027 - 001b: Skeleton Index 0
    0028 - 001c: Skeleton Index_Index 1
    0029 - 001d: NCX Index 0
    0030 - 001e: NCX Index 1
    0031 - 001f: NCX Index CNX
    0032 - 0020: FDST
    0033 - 0021: FLIS
    0034 - 0022: FCIS
    0035 - 0023: DATP
    0036 - 0024: EOF_RECORD

As you can see the FONTS can be stored *BEFORE* the image files, which means that I can not remove the FONT sections themselves without messing up all of the links to the images that are in the text records.

So the only way to create a type 7 mobi without FONTS is with Mobigen or a 1.X version of Kindlegen.

The best I can do is try keeping the font sections but making them 0 size. As I said earlier, I am not sure if that would even work. If you are willing to test things, I am willing to provide an experimental mobi_split.py that will walk the k7 part zeroing out any font sections but not actually removing the sections themselves.

KevinH

[KevinH]

Hi Everyone,

I have collected 3 fixes plus one experimental change to KindleUnpack and attached them here.

Included are:

kindleunpack.py - fix for multiprocessing on Windows

mobi_opf.py - all fixes to date plus hopefully all xmlescape
and unescape changes needed

mobi_split.py - fix for KindlePreviewer 2.8 and split mobi 7 files plus
an experimental fix to replace fonts with 0 size sections

All 3 can be found in the attached zip. I would appreciate hearing of any successes or failures.

Thanks,

KevinH

[adamselene]

Originally Posted by KevinH

Your font testcase is missing most of the fonts that are references in Section001.xhtml. So I am not sure how good a testcase this will turn out to be since kindlegen gives many warnings ...

I edited it rather hastily to remove anything that might possibly be unacceptable to distribute, and forgot to remove everything from the style sheet. Here's a fixed version.

In any case, I noticed that the super-secret hidden option in KindleGen 2.8 to output a M7/KF7 file does actually include the fonts. I have no idea why, since AFAIK they aren't used by *anything*.

[KevinH]

Hi,
Please try the updated pieces and see if zeroing out the section works unless you don't want it anymore. Either way, please let me know so that I can include the correct version in the upcoming release of KindleUnpack v62.

Thanks,

KevinH

Originally Posted by adamselene

I edited it rather hastily to remove anything that might possibly be unacceptable to distribute, and forgot to remove everything from the style sheet. Here's a fixed version.

In any case, I noticed that the super-secret hidden option in KindleGen 2.8 to output a M7/KF7 file does actually include the fonts. I have no idea why, since AFAIK they aren't used by *anything*.

[steppe]

Originally Posted by KevinH

Hi Everyone,

I have collected 3 fixes plus one experimental change to KindleUnpack and attached them here.

Included are:

kindleunpack.py - fix for multiprocessing on Windows

mobi_opf.py - all fixes to date plus hopefully all xmlescape
and unescape changes needed

mobi_split.py - fix for KindlePreviewer 2.8 and split mobi 7 files plus
an experimental fix to replace fonts with 0 size sections

All 3 can be found in the attached zip. I would appreciate hearing of any successes or failures.

Hi,

Just an update. When updated with the three patches, KindleUnpack v0.61 produces split .mobi files (Mobi7 and KF8) that work OK in the latest version of Kindle Previewer, 2.85, on all devices. In eInk Kindle mode, "Kindle" and "Kindle DX" devices, the emulations that are known to support only KF6 (Mobi7), the KF8 split file also works, but the ebook appears as a wall of text without any formatting. This is to be expected (not a bug). With the KF8-only file, the Kindle iOS mode is grayed out (disabled) most likely because "Kindle for iPhone" and "Kindle for iPad" emulations of Kindle Previewer v2.85 are outdated and do not yet support KF8.

[KevinH]

Hi,

Thanks for your test report. Here is an updated release of KindleUnpack_v62 with the fixes from updated_pieces.zip included.

- fix for improved command line handling on Windows boxes under multiprocessing
- improvements to split to allow it to work with newer Kindle Previewers
- improvements to split to allow it to replace fonts with zero-szed sections in Mobi 7
- improvements to opf to prevent html tags from interfering with opf xml creation

As before, we would welcome both reports of success and of course any failures reports.

Hope this helps,

KevinH

[pdurrant]

First post updated with the new version.