Kindlestrip Python script and AppleScript wrapper

[pdurrant]

Originally Posted by Xabache

Could someone write a step by step tutorial for this using kindlestrip. I have tried to follow along but fell flat on my face despite being generally knowledgeable of computers. Step by step please, download this, drag that... Thanks.

Download and install 32-bit (x86) ActivePython from here.
Download the kindlestrip 1.2.zip file. Unzip to give kindlestrip.py
Copy the book you want stripped to the same directory as kindlestrip.
Open a command window
use the cd command to switch to the same directory as kindlestrip
type python kindlestrip.py [BOOKNAME]
where [BOOKNAME] is the name of the ebook to be stripped.

[Peter Ahlstrom]

Paul, got a question. Is stripping actually necessary? I mean, when you upload an unstripped mobi to Amazon, does Amazon strip it themselves?

I had a mobi file made in kindlegen from an opf and html, and the unstripped version was 471 K. This was uploaded to Amazon and went on sale. Then I remembered I should strip it, so I did and got it down to 258 K. But then I looked at the version already on Amazon, and it said the filesize was 251 K.

So it seems to me that when the mobi was uploaded to Amazon in the first place, Amazon did something on the backend to already make the file as small as possible. Is there something I'm missing here?

(By the way, without mobiunpack I never would have been able to get the formatting looking very good. A million thanks for that.)

[pdurrant]

That's a very good question. And it seems that the answer, if the kindle file is destined to be uploaded for sale through Amazon, is no. Thanks for testing!

However, if one wants to distribute the kindle/mobipocket file oneself, perhaps by making it available for free on a web page, it might be useful.

On the third hand, I suppose the same could be done, apparently, by uploading the Amazon and downloading the test that Amazon creates for you, without ever actually making it available for sale.

[Peter Ahlstrom]

I'd be interested to see tests done, on how different a self-stripped file is from the one that Amazon generates. I can't test with my own file easily, since though I made it, the KDP account it's sold through is owned by someone else.

[zdavatz]

Kindlestrip does its full job for me. My file is back to 18 MB from 54 MB. How dumb of Amazon not keeping the option -nodatasource.

Best
Zeno

[DiapDealer]

It appears this won't work anymore for the new output generated by kindlegen v2.3.
I've been poking around with it (comparing code from the latest mobi_unpack), but I'm slow as molasses—plus I'm not really grasping exactly how to stitch the mobi back together minus the source section (but I can find the source section ).

Also, I was wondering if anyone else thinks it makes sense to have an option to strip (and save to a separate file) the old-mobi portion from the new mobi-KF8 hybrid? I still use kindlegen for my own personal use and won't always want the extra bulk of the additional KF8 data (especially when transferring to a device that can't make use of it).

[KevinH]

Hi,

It is not as easy as it looks for the new K8 versions. The damn SRCS record is no longer at or near the end of the .mobi and is in fact typically before the entire k8 mobi. So if we remove it from before the K8 part, you would really want to make sure that removing a section does not mess up any section references inside the K8 mobi part of the file.

Right now I think the K8 Mobi header references sections relative to the BOUNDARY record and I think all of the images and fonts come before the SRCS section.

If that is so, you may simply be able to copy the file section by section up to the SRCS section, omit it, and then update the starting table for all sections that come after it.

Kindlestrip now tries to look at the end to find the SRCS record so we would have to change that to walking though the sections 1 by 1 but it may work.

KevinH

Originally Posted by DiapDealer

It appears this won't work anymore for the new output generated by kindlegen v2.3.
I've been poking around with it (comparing code from the latest mobi_unpack), but I'm slow as molasses—plus I'm not really grasping exactly how to stitch the mobi back together minus the source section (but I can find the source section ).

Also, I was wondering if anyone else thinks it makes sense to have an option to strip (and save to a separate file) the old-mobi portion from the new mobi-KF8 hybrid? I still use kindlegen for my own personal use and won't always want the extra bulk of the additional KF8 data (especially when transferring to a device that can't make use of it).

[DiapDealer]

Originally Posted by KevinH

Kindlestrip now tries to look at the end to find the SRCS record so we would have to change that to walking though the sections 1 by 1 but it may work.

I've been able to walk the additional resources re-using some of the same code from mobi_unpack to consistently identify the section that contains the "SRCS" data type (in mobis generated with the new kindlegen and the old), but I'm weak on my prc/mobi specs and how the various offsets need to be adjusted when stitching the file back together. I'm using this as a learning tool, so someone will more than likely beat me to a solution—but that's the way it goes I guess.

[KevinH]

Hi,

If Paul does not beat us both to it, I will give it a go over the weekend.
The palm db header is pretty much the number sections in the database followed by a list of file starting offsets (and lengths?) and then followed up by those sections themselves.

If it helps, I believe Paul's Kindlestrip section code has the right code to adjust that table, it just needs to be modified to handle any record number as the SRCS record and to update all records until the actual end (since the SRCS need not and will not in K8s come at the or near the end).


BTW: we should be able to create something that removes the K8 part as well, but it may be complicated because of fonts and things stuck in the middle of the images. So instead of removing the sections completely (which would mess up the section numbers used in image links in the original mobi) we need to replace them with sections of 0 length.

Take care,

KevinH

Originally Posted by DiapDealer

I've been able to walk the additional resources re-using some of the same code from mobi_unpack to consistently identify the section that contains the "SRCS" data type (in mobis generated with the new kindlegen and the old), but I'm weak on my prc/mobi specs and how the various offsets need to be adjusted when stitching the file back together. I'm using this as a learning tool, so someone will more than likely beat me to a solution—but that's the way it goes I guess.

[KevinH]

Hi,

I made a quick and dirty kindlestrip_v130.py that does the job and creates a mobi that can be unpacked into exactly the same epub as before.

BUT ...

KindlePreviewer set to KindleFire will not properly show the new mobi document (it shows but all css is lost but if you look in the unpacked version it is properly there). So something someplace inside the K8 part of the real mobi is sensitive to us removing the SRCS section.

So until we find and figure that out, we will not be able to use it.

Perhaps I simply have a coding mistake but as far as I can tell (and as far as mobi_unpack.py can tell) this should work but does not.

I have attached the not truly working version. If anyone can see anything I am doing wrong, please let me know.

If the code is correct, then we might have some real work to do to get something working for K8 mobis.

[KevinH]

Hi,

I modified the above code to just replace the SRCS section with a section that has 0 length (and thereby not changed the number of entries in the palm db header) and obviously removed the SRCS data itself and now this version works with both KindlePreviewer and mobi_unpack.py.

So either the code above messes up that table somehow or something in the K8 side is pointing to a specific section number in the mobi and we need to simply replace the SRCS section with a section of 0 length to keep all section numbers the same.

If that is the case, I can't see it in the Palm Header anyplace!
If anyone wants to see this modified version just let me know.

KevinH

[DiapDealer]

Originally Posted by KevinH

Hi,

I modified the above code to just replace the SRCS section with a section that has 0 length (and thereby not changed the number of entries in the palm db header) and obviously removed the SRCS data itself and now this version works with both KindlePreviewer and mobi_unpack.py.

So either the code above messes up that table somehow or something in the K8 side is pointing to a specific section number in the mobi and we need to simply replace the SRCS section with a section of 0 length to keep all section numbers the same.

If that is the case, I can't see it in the Palm Header anyplace!
If anyone wants to see this modified version just let me know.

KevinH

I'd like to see the modified version. I've tried to make my own changes to the v1.3 code—based on what you've described—but my final product is crashing the new Kindlepreviewer app... so it's safe to assume I have something wrong.

[JSWolf]

Originally Posted by KevinH

K8

It's KF8, not K8.

[KevinH]

Originally Posted by DiapDealer

I'd like to see the modified version. I've tried to make my own changes to the v1.3 code—based on what you've described—but my final product is crashing the new Kindlepreviewer app... so it's safe to assume I have something wrong.

Hi DiapDealer,

Try this one ...

But I have not tried it with anything other than the KindlePreviewer and the Jerome.mobi with the SRCS section stripped but the entry in the table kept but set to 0 length.

I have no idea if this will mess up any other reading software (including Calibre!)

Take care,

KevinH

[KevinH]

Hi DiapDealer,

At least I know why stripping out the SRCS section causes problems. I could not figure out where in the palm db or first mobi header they could store information that indicates where the K8 version started without having to manually load and examine each section looking for the boundary. And everything I can find in the K8 header is relative to its own starting section number.

The only place they could add info and not mess up the old mobi format was if they stored the extra info in new metadata fields.

And sure enough, that is what they did!

If you look in the old mobi's *.opf you will see unknown metadata key 121 whose value is the section number of the BOUNDARY section! You will also see a metadata key 129 that is actually the ascii string "kindle:embed:0001" and metadata key 125 which appears to be a count of something and metadata key 131 which always seems to be 0.

If you look in the K8 *.opf file you will see they have removed metadata key 121, metadata key 129 is still "kindle:embed:0001", metadata key is 131 is still 0 and now metadata key 125 has been set to 0.

So if we actually remove the SRCS section (it is the last section before the BOUNDARY section), then we need to change at least metadata key 121 in the old mobi exth data to point to the new boundary section. We should also figure out what the other new metadata values are used for just in case.

If I get time over the weekend, I will revert to the previous kindlestrip_v130.py and try to have it fix the metadata 121 value in the old mobi header to reflect the removal of the SRCS and see if that does the trick.

This does explain why simply zeroing the size of the SRCS entry works, it does not change the section number of the BOUNDARY SECTION.

Take care,

KevinH