[GUI Plugin] KindleUnpack - The Plugin

[Tugger]

About the licensed font issue - InDesign decided to embed them by default (I could have asked it not to). So presumably it is OK to distribute them? Perhaps I need to read the Adobe small print.

[pdurrant]

Originally Posted by Tugger

About the licensed font issue - InDesign decided to embed them by default (I could have asked it not to). So presumably it is OK to distribute them? Perhaps I need to read the Adobe small print.

If you intend to distribute the book, you do indeed need to read the fine print in your font licences.

[DoctorOhh]

Originally Posted by Tugger

That is very very strange. Same operating system (Win 7 64bit), same version of Calibre, same version of plugin, same file to unpack. And I tried closing Calibre then running it again with the same result. And I'm only using about 3GB of the 12GB in the machine. Now what do I do?

In Windows I believe calibre is a 32 bit application so you won't be able to use more memory. Have you rebooted your computer?

[DiapDealer]

@Tugger: Is there any chance that you have another plugin installed—perhaps an experimental version of this one—that uses the same plugin-import-name? I know it's a long shot, but I'm struggling for possibilities, here.

[theducks]

Originally Posted by DiapDealer

@Tugger: Is there any chance that you have another plugin installed—perhaps an experimental version of this one—that uses the same plugin-import-name? I know it's a long shot, but I'm struggling for possibilities, here.

Or one of those we can't mention here at MR

[DiapDealer]

Originally Posted by theducks

Or one of those we can't mention here at MR

Even that really shouldn't be an issue.

I was just thinking that if there were another plugin installed with the same plugin-import-name (the dummy text file) that also imported mobi_unpack.py—you know, a quick, copy/paste, change-the-plugin-name-and-think-I'm-all-done type experiment—perhaps a different version of mobi_unpack.py is being imported into this plugin and causing failures.

It's a long-shot, I know, but I'm guilty of that all the time myself—testing/debugging different versions of plugins. I make a new plugin, but forget to change that dummy text-file's name that handles the importing of other python modules and then it takes me forever to figure out that I'm actually importing a module from an entirely different plugin!

Probably not the issue here, but I thought I'd throw it out there just in case.

[Tugger]

Thanks for all the suggestions.

Yes, Calibre is a 32 bit application. But under Windows 64 bit, each 32 bit application gets its own 32 bit address space, so it is very unlikely indeed to run out of memory. Far less likely than on a 32 bit operating system.

I have only 155 books in my Calibre collection.

I have only one other plugin. It is a popular file type plugin, let's say no more. I have never had any other plugin installed apart from earlier versions of the unpack one, including the last beta.

I just rebooted my computer, although this does not seem a logical thing to do. Result: it just unpacked it with no problems at all. And I carried on unpacking the file six times in all with no problem at all.

Nice to know that when desperate to Mobiunpack I can reboot.

But something is wrong somewhere.

[DiapDealer]

Originally Posted by Tugger

But something is wrong somewhere.

I (respectfully) suspect your computer or your calibre installation. That's where all signs seem to be pointing.

I've debugged the plugin on three platforms (both 32- and 64-bit) on countless different machines, both real and virtual (all with nowhere near 12Gb of ram), and calibre 0.8.41 to present. I've encountered scads and scads of error messages during that process, but I never once got the memory error you're getting. Nor have I had any other user reports of a similar memory error.

Did that same error ever show up again the previous time? From the time you reported it started working until this latest update, I mean?

[Tugger]

Originally Posted by DiapDealer

Did that same error ever show up again the previous time? From the time you reported it started working until this latest update, I mean?

No, everything has been fine. And now everything is fine again.

My computer is rock solid and runs a variety of heavy duty software (mostly photo editing related) without any problems at all, month after month. It only gets rebooted after the monthly Microsoft update which asks for the reboot.

Could there be some weird thing where a reboot is needed after installing a new version of Calibre or the plugin?

Thinking about it, the previous time I had the problems was at another location on my other main computer, which is a different motherboard etc but similar power, an i7 950 with 12GB in that case.

[pdurrant]

Perhaps it's some kind of caching problem.

[Tugger]

Please can you tell me how to reverse the effect of running the plugin. In other words how do I take all the folders it has made, with their files, and turn it all back into one mobi file? I need mobi-pack.

The reason is that I have used mobi-unpack to investigate what the Amazon InDesign plugin is doing wrong with its KF8 version. I have now pretty much worked this out, to correct it involves some manual edits to the styles. So I need to be able to take the mobi7 and mobi8 folders, which contain the two versions, and put them together again.

I do have some worries about what I'm doing. I can make the KF8 mobi file from the mobi8 folder contents and it is now OK. But the top level of that folder contains an epub file, so I'm hoping that KDP don't use that, as it will still be wrong. Also they put a file called kindlegensrc.zip in there, and I'm wondering if I really need to give that to kdp. If so I would have to fix the css file in there too. The problem is that I don't know what the KDP system actually does when given the combined mobi with its three versions. The kindlegensrc version seems to relate only to the mobi8 folder which contains the KF8 version. Perhaps it is just there for debugging purposes.

When submitting a handmade mobi file to kdp, when that is a KF8 submission, are people supposed to also submit the separate old format file as well, as in the generated mobi I showed you? Or does KDP create the old version automatically from the KF8 version? (If this is the case I don't need to pack it all back together, I can simply upload my new KF8 mobi file).

Sorry to ask these questions here, as they are straying from simply packing and unpacking, none of this seems to be made clear enough by Amazon.

[Tugger]

After more research and experiments let me try to answer my own questions and perhaps you will be kind enough to tell me if I'm getting this right.

The InDesign mobi file contains a mobi7 folder with the old format files. A mobi8 folder with the KF8 format files and a kindlegensrc.zip file.

I can unpack it all, and go to work fixing the problems with the contents of the mobi8 folder. I can change the styles for example.

I can then take the opf file from the mobi8 folder and drop it onto the Previewer, which is equivalent to running Kindlegen from a command line. I can take the resulting compiled mobi file and put it into Calibre. I can delete all the files from the previous unpacking. I can then unpack the new file and by magic I see that I again have everything I had before, the mobi7 folder is back. So the answer is that it is Kindlegen which will automatically read a KF8 set of files, including styles etc, and it creates not only the KF8 but also the KF6 (mobi7) version, all at once in the new mobi file.

In fact I realise that the InDesign Plugin itself uses Kindlegen.

Anyway, all I have to do with this workflow is run the plugin, load the result, unpack it, keep only the mobi8 folder, edit the files as needed, compile using Previewer and submit the result to KDP.

There are two problems with the plugin. One is to do with setting the start position for goto beginning. I have not quite cracked that one yet. The far more serious one is that InDesign uses points to specify font sizes, which is perfectly natural. But the plugin fails to convert these to, for example, ems. The result is very small fonts and the user can't make them large enough for a comfortable read, except by going to the 7th size when they suddenly become far too large. Using ems in the styles solves this.

So all this means I don't need a mobi-pack plugin after all.

Hope I'm not boring you with all this.

[pdurrant]

Originally Posted by Tugger

Anyway, all I have to do with this workflow is run the plugin, load the result, unpack it, keep only the mobi8 folder, edit the files as needed, compile using Previewer and submit the result to KDP.

Exactly so.

Once I have a moderately good source folder, I only use unpack to see exactly what KindleGen is up to — I just keep going back to my same source folder and altering that and running it through KindleGen again. (I use the Drag&Drop AppleScript wrapper for KindleGen, and keep it on 'fast' until I'm happy with the result, and then compile as 'small'.)

[DiapDealer]

That's mainly how I do it, too.

A word of caution, though: if you're using the contents of the mobi8 folder (created by mobi_unpack) to reconstruct your MOBI using Kindgen/Previewer, you're building the joint MOBI/KF8 file from only the KF8 portion of the source. Meaning that if you originally created a joint MOBI/KF8 that uses media-queries to manage (compensate for) the incompatibilities between the two formats, you're going to lose the original MOBI 7 portion (I mean it will be built using only the KF8 side of those media-queries, not that the MOBI 7 portion won't be there), when you rebuild the mobi8 folder contents

Because of that, I rarely use the mobi_unpack output to rebuild the ebook (unless it's something really simple where there were no media-queries involved in the original). I mostly use mobi_unpack for inspection and then return to the original to make the changes.

I realize that the original being an InDesign project might hamper that ability to return to the original source, though. Does the ID plugin include the "source" as part of its final product? If so, that's what I'd return to change/rebuild. But I don't even know if the ID plugin utilizes the media-queries (behind the scenes) to tweak any differences between the two portions of the joint file—so this could all be moot. The bottom line is, of course, if you're happy with your workflow and the final output you're producing. Then keep doing what you do.

[Tugger]

Very helpful comments, thank you. Yes, the output mobi file from the plugin includes a file called kindlegensrc.zip, which I now understand is the "source". That is what I should be editing, not the files in the mobi8 folder. I've just looked at the template.css file in the source and it does have a couple of media queries in it. Can media queries also be in other non .css files?

So mobi-unpack is crucial, as it is the way I can get the "source" out of the generated mobi file.

My solution to the font size problem is working very well. I make the standard paragraph font in the document 10 point. Then I go through the generated css file searching for "pt" and each time I replace pt with em and divide the number before it by 10. So 30pt become 3.0em.

Now I shall tackle the missing start feature. The plugin doesn't support it at all, so I need to add it to the "source" before recompiling.

I recommend InDesign despite my problems. It would be perfectly possible to make a beautiful printed book with lots of typographical complexity and then make a well formatted Kindle edition quickly with no html or css knowledge at all. It handles fonts, including embedded fonts if wanted, (even if just used for bullets), TOC, NCX file automatically generated, and the cover. And it covers pretty much all the formatting that is possible with KF8.