Kindlestrip Python script and AppleScript wrapper

[pdurrant]

Originally Posted by tomsem

This will be especially useful now that the Personal Documents service preserves kf8 formatting. AFAICT, it does not strip the source before storing (file size is only modestly reduced), and there is a 50MB maximum file size enforced by Send to Kindle (and that limit is higher than can generally be sent as an email attachment). In some cases, it might be helpful to upload separate kf8 and mobi files to get under this minimum (mobi still required for iOS). Is there a way to do this?

KindleUnpack can split a file into separate Mobi and KF8 files.

[tomsem]

Originally Posted by pdurrant

KindleUnpack can split a file into separate Mobi and KF8 files.

Thanks,!

[steppe]

Originally Posted by pdurrant

As always, please comment with any bug reports or problems.

Hi. This is probably not a bug or problem, I just noticed that the stripped mobi from KindleStrip1.35 is slightly smaller than the stripped file produced by KDP, about 18KB difference. Why could this be?


Update: I used KindleUnpack to investigate and it turns out that this difference in size is due to an extra JPEG thumbnail of the cover image added by KDP (180 x 240 pixels, 17.3KB). It's a puzzle why KDP would do that because an identical thumbnail is already present in the ebook. KindleStrip doesn't add that extra thumbnail. I guess I answered my own question

[pdurrant]

OK, I've downloaded and unpacked both with KindleUnpack 0.6.1., with the log everything option.

The stripped by KDP (presumably you mean uploaded to Amazon and this is the actual published file) has one more section in the Mobi file format, which results in 8 more bytes in the Mobi header.

Interestingly, it has the header fields identified as "first_content" and "last content" set to -1, instead of 1 and 61. I doubt that that's significant.

The extra section seems to come immediately after section 61 (the last of the text sections).

The KDP one also has more EXTH metadata - an extra 12 bytes, which is an entry for K8_Boundary_Section, which is set to -2 (0xFFFFFFFE).

The KDP one also has 12 fewer bytes (of NULLs) at the end of the header. 8180 instead of 8192. This is probably because of the extra EXTH.

Aha! The extra section, section 62 is an extra image file. It seems to be another Thumbnail image of the cover. Which is odd, as that's already present as section 61. The section 62 version is very nearly the same length, but has different binary coding. i.e. it's not an exact copy of 61, but probably newly scaled version of the cover image.

The extra image section is 0x456C (=17772) bytes, and there's the 8 bytes for the index to the section. That's a total of 17780 bytes in the KDP file.

1098644 + 17780 = 1116424

I wonder why KDP adds an extra thumbnail image.

(Checking in more detail, in the KDP version the EXTH identifying the thumbnail image does point to the new section. But in the other one, the existing thumbnail image is correctly identified, so KDP should know that it was there. Very mysterious.)

[steppe]

Originally Posted by pdurrant

I wonder why KDP adds an extra thumbnail image.

You just stole my thunder as I reached a similar conclusion myself Thanks for investigating this stuff and for the detailed response. Much appreciated.

The original file with the source attached was made by KindleGen 1.2 (old version) and perhaps this is the reason why KDP tries to provide an updated cover thumbnail.

Update: Yes, this was probably due to the old version of KindleGen because if I build the ebook using the latest KindleGen (2.8) and upload that file to KDP, the extra thumbnail is not added. Interestingly, KDP fails to strip the source files this time around (the .mobi file in the "Downloadable Previewer" still contains the attached source files), but if I proceed to the next publication step (Pricing and rights), KDP does show the reduced size of the file that will be published (1.05MB versus 4.84MB). Unpacking the .mobi file from the "Downloadable Previewer" (using KindleUnpack.61) suggests that the final published file is nearly identical in size to the file produced by KindleStrip: 4,965KB minus 3,827KB zipped source equals 1,138KB. The file produced by KindleStrip1.35 is 1,136KB.

[jefftriplett]

Since a few of my side-projects relies on Kindlestrip I've been maintaining a version on github for a while. It's also available on pypi via `pip install kindlestrip` for anyone who uses pip.

[adam_omega]

I code comic books and manga for the various eReader devices and am finding that if I use "KindleGen v2.9 for Windows (XP, Vista, 7)" and then strip the file with "kindlestrip.py 1.35"...
The end of the books show repeated pages and pages that are out of order (sometimes even the book's cover repeated). If I try to upload this data to KDP, I get an invalid mobi file error.

[Update: the problem seems to be with KindleGen v2.9. Downgrading to v2.8 doesn't have the issue when running KindleStrip.]

[pdurrant]

Originally Posted by adam_omega

I code comic books and manga for the various eReader devices and am finding that if I use "KindleGen v2.9 for Windows (XP, Vista, 7)" and then strip the file with "kindlestrip.py 1.35"...
The end of the books show repeated pages and pages that are out of order (sometimes even the book's cover repeated). If I try to upload this data to KDP, I get an invalid mobi file error.

[Update: the problem seems to be with KindleGen v2.9. Downgrading to v2.8 doesn't have the issue when running KindleStrip.]

Thanks for the report. If you could send me (paul@durrant.co.uk) source files that show the problem, I might be able to take a look at some point in the next couple of weeks.

Note that if you're uploading to Amazon for KDP, there's no need to use KindleStrip unless the pre-kindlestrip files are above Amazon's file upload limit.

[adam_omega]

Originally Posted by pdurrant

Thanks for the report. If you could send me (paul@durrant.co.uk) source files that show the problem, I might be able to take a look at some point in the next couple of weeks.

Sure, I'll send you an email some file links next week.

Note that if you're uploading to Amazon for KDP, there's no need to use KindleStrip unless the pre-kindlestrip files are above Amazon's file upload limit.

Yes, they're 160~200 page books that clock in at around 40mb each on their own (as they're nothing but image files), so they have to be striped to be uploaded.

[steppe]

Originally Posted by adam_omega

I code comic books and manga for the various eReader devices and am finding that if I use "KindleGen v2.9 for Windows (XP, Vista, 7)" and then strip the file with "kindlestrip.py 1.35"...
The end of the books show repeated pages and pages that are out of order (sometimes even the book's cover repeated). If I try to upload this data to KDP, I get an invalid mobi file error.

[Update: the problem seems to be with KindleGen v2.9. Downgrading to v2.8 doesn't have the issue when running KindleStrip.]

Interesting. I tested my fixed-layout templates and was unable to reproduce this issue. Built using KindleGen 2.9 (Windows) and tested in Kindle Previewer 2.901 (Fire, Fire HD, Fire HD 8.9", and Paperwhite modes) and on my Paperwhite device. The templates are about 14 pages long, 600 x 1024 pixels; I tested both the "comic" book type, portrait orientation (with or without a TOC page) and the "children" book type, landscape orientation (with or without a TOC page). Attaching the source files.

[quiris]

I have problem with stripped Mobi file.

1. The file w-obronie-wolnosci.epub (created with Sigil) has got logo "helion.pl" placed on title page.
2. The file w-obronie-wolnosci.mobi was created with kindlegen 2.9 for Mac.
3. The file w-obronie-wolnosci_stripped.mobi was created with KindleStrip 1.35 and has got following problem:

a. The file loaded in KindlePreviewer 2.92 with KindleFire settings has got replaced logo "helion.pl" with other image.
b. The same file loaded in KindlePreviewer 2.92 with Kindle e-ink settings has no such problem.

[pdurrant]

Originally Posted by quiris

I have problem with stripped Mobi file.

1. The file w-obronie-wolnosci.epub (created with Sigil) has got logo "helion.pl" placed on title page.
2. The file w-obronie-wolnosci.mobi was created with kindlegen 2.9 for Mac.
3. The file w-obronie-wolnosci_stripped.mobi was created with KindleStrip 1.35 and has got following problem:

a. The file loaded in KindlePreviewer 2.92 with KindleFire settings has got replaced logo "helion.pl" with other image.
b. The same file loaded in KindlePreviewer 2.92 with Kindle e-ink settings has no such problem.

Can you confirm that the output of KindleGen (before kindlestrip) displays correctly? It is not clear from your post that you have checked this.

[KevinH]

Hi Paul,

Please try the attached DumpMobiHeader_v014.py on a Kindlegen 2.9 generated mobi. You will see that there is now an extra BOUNDARY followed by a CONT section (container) that has its own exth and which has following sections where they are keeping high definition format copies of specific images.

I have modified DumpMobiHeader_v014 to at least show those new sections and dump the exth values in the new CONT header. There are other values in the CONT header as well but I am not sure what any of them are yet.

My guess is that the Kindle Fire and HD use these high resolution images stored outside the mobi 6 part and that somehow when kindlestrip removes the SRCS sections it is throwing off the code that accesses these now shifted high definition images.

My rec would be to go back to nulling out the SRCS sections so that we don't upset any offsets to images we do not understand yet.

I have attached the latest DumpMobiHeader_v014.zip here so please look at the original mobi using it and see if that makes any sense to you.

Take care,

KevinH

[quiris]

Originally Posted by pdurrant

Can you confirm that the output of KindleGen (before kindlestrip) displays correctly? It is not clear from your post that you have checked this.

I'm absolutely sure (200%). Original file displays the logo correctly.

[pdurrant]

Originally Posted by KevinH

My rec would be to go back to nulling out the SRCS sections so that we don't upset any offsets to images we do not understand yet.

I have attached the latest DumpMobiHeader_v014.zip here so please look at the original mobi using it and see if that makes any sense to you.

Thanks, Kevin. It looks like an update is needed, but I don't know when I'll get time.