Kindlestrip Python script and AppleScript wrapper

[dilo_sec]

Being naturally curious, I also dumped the pagemap section of my 2nd ebook - the start looks the same (these could be fixed values), the value at 0x0038 looks to denote the number of bits, 0010 (16-bit) or 0020 (32-bit), for the page offset values.

A dump of the pagemap is below, including part of the page-map.xml source - I've not figured what the text following "pageMap": means - it would not be needed during mobi unpacking to generate a page-map.xml.

-- start
0x0000: 50414745 PAGE

0x0004: 00000008 ?
0x0008: 00010001 ?
0x000C: 0000002A ?

0x0010: 0000 block 0?
0x0012: 001E size of block, 0x0032-0x0014 = 1E

0x0014: 7B0A {
0x0016: 2020 20226669 ... "fileRevisionId" : "1"
0x0030: 7D0A }

0x0032: 0001 block 1?
0x0034: 0079 size of block, 0x00B3-0x003A = 79
0x0036: 0183 pages in pagemap
0x0038: 0020 bits is page offset values

0x003A: 7B0A {
0x003C: 2020 20226465 ... "description" : "PageMap from source by kindlegen",
0x0073: 2020 20227061 ... "pageMap": "(1,c, |i),(3,r,2),(7,r,7),(10,r,11),(12,a,1)"
0x00B1: 7D0A }

0x00B3: 00000028 cover
0x00B7: 000025C0 i
0x00BA: 000028D5 ii
0x00BE: 00002C0F iii
<snip>
-- end pagemap

-- start
<page-map xmlns="http://www.idpf.org/2007/opf">
<page name=" " href="cover.html"/>
<page name="i" href="frontm.html"/>
<page name="ii" href="frontm.html#page_ii"/>
<page name="iii" href="frontm.html#page_iii"/>
<page name="iv" href="frontm.html#page_iv"/>
<page name="v" href="frontm.html#page_v"/>
<page name="vii" href="frontm.html#page_vii"/>
<page name="viii" href="frontm.html#page_viii"/>
<page name="ix" href="frontm.html#page_ix"/>
<page name="x" href="frontm.html#page_x"/>
<page name="xi" href="frontm.html#page_xi"/>
<page name="xii" href="frontm.html#page_xii"/>
<page name="1" href="chapter01.html"/>
<page name="2" href="chapter01.html#page_2"/>
<page name="3" href="chapter01.html#page_3"/>
<page name="4" href="chapter01.html#page_4"/>
<page name="5" href="chapter01.html#page_5"/>
<snip>
-- end page-map.xml

[KevinH]

Hi,

Wow! Nice work indeed!

You are taking almost all of the fun out of it by figuring things out before anyone else has even had a chance to stare at it yet.

Thanks for leaving a few unknowns for the rest of us! ;-)

KevinH

Originally Posted by dilo_sec

KevinH,

The pagemap section maps the page numbers to offsets into the raw (uncompressed) html in the mobi file - (only) the offsets are different for Mobi6 and Mobi8 parts.

I've decoded the pagemap section as follows: (the use of some values are unknown?)

-- start
0x0000: 50414745 PAGE

0x0004: 00000008 ?
0x0008: 00010001 ?
0x000C: 0000002A ?

0x0010: 0000 block 0?
0x0012: 001E size of block, 0x0032-0x0014 = 1E

0x0014: 7B0A {
0x0016: 2020 20226669 ... "fileRevisionId" : "1"
0x0030: 7D0A }

0x0032: 0001 block 1?
0x0034: 0054 size of block, 0x008E-0x003A = 5A
0x0036: 000A pages in pagemap
0x0038: 0010 ?

0x003A: 7B0A {
0x003C: 2020 20226465 ... "description" : "PageMap from source by kindlegen",
0x0073: 2020 20227061 ... "pageMap" : "(1,a,1)"
0x008C: 7D0A }

0x008E: 0281 page_1
0x0090: 0500 page_2
0x0092: 074A page_3
0x0094: 0959 page_4
0x0096: 0B49 page_5
0x0098: 0DCA page_6
0x009A: 1014 page_7
0x009C: 1176 page_8
0x009E: 1413 page_9
0x00A0: 164E page_10
--end pagemap

Apologies about the formatting - all the tabs and mutiple spaces have been scrunched up to 1 space!

Hope this is useful ...

DS.

[KevinH]

Hi DS,

You wrote:

0x003A: 7B0A {
0x003C: 2020 20226465 ... "description" : "PageMap from source by kindlegen",
0x0073: 2020 20227061 ... "pageMap": "(1,c, |i),(3,r,2),(7,r,7),(10,r,11),(12,a,1)"
0x00B1: 7D0A }

0x00B3: 00000028 cover
0x00B7: 000025C0 i
0x00BA: 000028D5 ii
0x00BE: 00002C0F iii
<snip>
-- end pagemap

-- start
<page-map xmlns="http://www.idpf.org/2007/opf">
<page name=" " href="cover.html"/>
<page name="i" href="frontm.html"/>
<page name="ii" href="frontm.html#page_ii"/>
<page name="iii" href="frontm.html#page_iii"/>
<page name="iv" href="frontm.html#page_iv"/>
<page name="v" href="frontm.html#page_v"/>
<page name="vii" href="frontm.html#page_vii"/>
<page name="viii" href="frontm.html#page_viii"/>
<page name="ix" href="frontm.html#page_ix"/>
<page name="x" href="frontm.html#page_x"/>
<page name="xi" href="frontm.html#page_xi"/>
<page name="xii" href="frontm.html#page_xii"/>
<page name="1" href="chapter01.html"/>
<page name="2" href="chapter01.html#page_2"/>
<page name="3" href="chapter01.html#page_3"/>
<page name="4" href="chapter01.html#page_4"/>
<page name="5" href="chapter01.html#page_5"/>



So ...

"pageMap": "(1,c, |i),(3,r,2),(7,r,7),(10,r,11),(12,a,1)"

must in some way code up information in the "preface" pages up to the point where normal page numbering begins.

Do you have a sample that includes appendices or bibliography numbered in a different scheme (maybe A-1 or A2), etc that might show us more?

If not, we can always cook up a few samples that we might be able to use to figure out the exact meaning and construction of:

"pageMap": "(1,c, |i),(3,r,2),(7,r,7),(10,r,11),(12,a,1)"

I am thinking here more along the lines of getting calibre to generate the same information when it creates joint or azw3 mobis.

Also perhaps we need to rename what we call the SRCS offset in the header to be more generic. Perhaps something along the lines of an offset to additional pieces that need not be part of the mobi itself (PAGE, SRCS, and ...?), so perhaps ADDONS offset and count might be a more appropriate way of thinking about this header field in both the Mobi6 and Mobi8 headers.

Thanks again for finding and figuring this out. Patches and changes for Mobi_Unpack are of course welcome! ;-)

Take care,

KevinH

[SusanM]

Originally Posted by pdurrant

Kindlegen 1.1 and later now always adds the source files used in compiling the kindle ebook as the penultimate record in the kindle ebook.

So I wrote a python script that strips out the sources record from Kindle format ebooks. And for those on Macs I wrote a nice Applescript wrapper and also put the python script in the AppleScript bundle to make things easy.

If you're going to upload to the Amazon store, this script is probably unnecessary, as Amazon will strip the sources before delivery anyway.

But if you're going to upload the ebook somewhere else, (e.g. the MobileRead Library), you might well want to make the file as small as possible.

KevinH has enhanced the original python script to handle newer Kindle format ebooks correctly.

Here they are. A python script, and an AppleScript wrapper around it. If you're on a Mac you only need the Applescript, as it includes the Python script in it. The Applescript is a simple drag&drop operation — drag your KindleGen generated file onto it, and it creates one named [oldname]_stripped.mobi.

As always, please comment with any bug reports or problems.

Great script and thanks!

[SusanM]

I have tried both scripts for Kindlegen conversion and the KindleStripper posted by pdurrant. Nice addition to eBook tools.

[Skeezix]

Looks like something changed with kindlegen 2.7. Using kindlestrip on a MOBI generated with 2.7 corrupts the file, so that my Kindle Keyboard cannot read it. The stripped MOBI I generated with 2.5 (using the same opf) works fine.

[DiapDealer]

No clue there was even a 2.7 kindlegen out there.

[pdurrant]

Originally Posted by Skeezix

Looks like something changed with kindlegen 2.7. Using kindlestrip on a MOBI generated with 2.7 corrupts the file, so that my Kindle Keyboard cannot read it. The stripped MOBI I generated with 2.5 (using the same opf) works fine.

Thanks for the note. Time to take a look at KindleGen 2.7!

[pdurrant]

Now fixed. First post updated.

KindleGen 2.7 adds an additional record that stores the output from KindleGen. KindleStrip almost took account of this, but failed on one line. KindleStrip 1.35 now handles this correctly, and this extra record also gets stripped out properly.

[Skeezix]

Thanks for fixing that so quickly!

[steppe]

Originally Posted by pdurrant

Now fixed. First post updated.

Thank you from the bottom of my heart for this useful tool. I compiled a fixed layout ebook with about 200 pages of photos and the file size with the source attached was over 80MB. KDP doesn't allow one to upload files bigger than 50MB, but the thing is my file is only 33 MB without the source files attached by KindleGen 2.7. So if your file plus the source is bigger than 50MB there is no way you can use KDP to remove the source attachment. Thanks to KindleStrip I removed the attachment and uploaded the 33MB file without any problems. The resulting fixed layout ebook works great both in the "Online Previewer" and in Kindle Previewer.

[SusanM]

Thank you for sharing!

[jayz]

How do you use it on a mac?

[pdurrant]

Originally Posted by jayz

How do you use it on a mac?

Download the AppleScript application.. (KindleStrip 1.35.app.zip)

Unzip it by double-clicking on the zip file, if your browser hasn't already unzipped it for you.

Drag an ebook created by KindleGen onto the AppleScript application.


In general, only use it for books you're distributing yourself, not for books uploaded to KDP. Steppe's situation was unusual.

[jayz]

"not for books uploaded to KDP"
Why? Can you explain.