Tutorial - How to Create a MobiPocket eBook

[tompe]

I understand you practical viewpoint. I just think that you should not support secret or proprietary format by using them if you can aviod it. That might put some pressure on companies to release details about their format.

And people putting books on MobilRead should definitely avoid to use the high compression format.

[cmbs]

Originally Posted by DMcCunney

I have a query in at the MobiPocket forums about whether documentation for their algorithm exists.

Frankly, I'm torn. I understand your concern, but storage space is finite.

Most of my ebooks are in Plucker format, and I always use the High Compression option, which uses a Palm port of ZLib and provides zip/gzip compatible compression instead of the standard Palm "doc" compression.

If I didn't do that, I'd have had to upgrade from a 2GB SD card to a 4GB card long ago for the volume that holds my Plucker documents. (I have about 3,200 Plucker files occupying about 1.5GB on that card.)

Using High Compression in MobiPocket becomes an issue if someone wants to use MobiPerl to manipulate the files because your code can't uncompress them.

I'm mainly concerned that anyone can read a Mobi file I create. The number of folks who might want to use MobiPerl to manipulate them will be a small subset of the total number who might read them.

For stuff I create for myself, I'd likely use High Compression. For stuff I upload elsewhere, it's a quandary, and I'm thinking.
______
Dennis

I use high compression. The files are perfectly readable on the desktop reader and my cybook. Nobody has said anything about not being able to read the files I've posted. I will continue to use high compression, I don't see any reason to have larger files than necessary, I prefer them small as I can get them.

[Jiiri]

I think tompe is talking about files created with high compression not being able to be 'unpacked' or otherwise manipulated by his Mobiperl scripts. Since a lot of people (I am one) like to manipulate the metadata after we download a file, any file for distribution should be created using normal compression.

I must admit not really getting the issue anyway - these files are tiny. Hard drive space is about as cheap as it gets, and books are tiny. My Kindle has about 180 MB of free on-board memory and a 2GB SD card, and it holds approx 2200 books.

Given that the files are small, and the fact that many people like to manipulate the metadata for viewing on their ebooks readers, I would imagine that normal compression is optimal. You could always zip the files for storage if it's much of an issue.

Jiiri

[DMcCunney]

Originally Posted by Jiiri

I must admit not really getting the issue anyway - these files are tiny. Hard drive space is about as cheap as it gets, and books are tiny. My Kindle has about 180 MB of free on-board memory and a 2GB SD card, and it holds approx 2200 books.

Jiiri

Hard drives are huge and cheap. Storage on handhelds is another matter.

My device is a Tapwave Zodiac 2, which is a PalmOS PDA. It has two SD card slots, and a 2GB card in each.

My main ebook reader is Plucker, and I create Plucker files using the high compression option, which uses a Palm port of Zlib (supplied with Plucker) to decompress. I have about 3,200 Plucker volumes on the first SD card, occupying about 1.5 GB. If I used standard "doc" compression, I'd have to either remove books, spread the Plucker library over more than one card, or switch to a 4GB card. (The first card holds installed applications that live on the card and a variety of other things besides Plucker files.)

My Mobi files are on the second card, which is mostly occupied by videos played with TCPMP. There isn't a lot of free space there, either.

4GB cards are in my future, but not until I don't have another option. If I can use high compression to save space, I do so.

And as it is, I'm lucky. The Zodiac takes 2GB cards out of the box, and the installation of a FAT32 driver ripped from a LifeDrive lets it handle 4GB cards. (It does not handle SDHC cards, so higher capacities aren't possible.)

There are people readign ebooks on older Palm devices that don't handle cards larger than 1GB, and people running things like Sony Clies which use Memory Sticks with a maximum capacity of 128MB. Guess how enthusiastic they are about files larger than they have to be?

Yes, I'm a pack rat. I like to have my entire electronic library on my device, and not have to select a subset because there isn't room for everything.

My inclination is to create the smallest ebook file supported by the reader, for benefit of folks who feel the same way and may have less available space to hold things.
______
Dennis

[Jiiri]

That makes sense. My apologies if my post in any way sounded other than constructive.

The problem only arises if you do a lot of work on a book, say, from PG, and want to share your work with other people on these forums. In that case, since you have the source files, you can create a normal compression version for public consumption and a high compression one for yourself.

I, of course, have no problem at all with anyone saving their own files however they like. If I understand these posts correctly, the only problem is that tompe's superb MobiPerl scripts won't work with high compression files. At this point it seems that something has to give - either use normal compression, or don't have access to tompe's MobiPerl tools, I guess.

And I can relate - I'm a pack-rat myself.

Jiiri

[DMcCunney]

Originally Posted by Jiiri

That makes sense. My apologies if my post in any way sounded other than constructive.

No apologies needed. It's a question that can be argued either way.

The problem only arises if you do a lot of work on a book, say, from PG, and want to share your work with other people on these forums. In that case, since you have the source files, you can create a normal compression version for public consumption and a high compression one for yourself.

Or create a high compression version for public consumption, with an offer to spin a no compression or standard compression version for folks who wish it for one reason or another.

Most folks just want to read Mobi titles, hence my original question about whether the high compression option was known to work on all platforms that handled Mobi files. The number who want to manipulate them with MobiPerl will be a much smaller number, and the sorts of things they might want to do may be corrections of the "Yeah, you're right. It should have been like that to begin with. I'll create and post a corrected edition." once I'm aware of the problem. For that matter, I'd have no problem with providing my source files for people to create their own versions.

I, of course, have no problem at all with anyone saving their own files however they like. If I understand these posts correctly, the only problem is that tompe's superb MobiPerl scripts won't work with high compression files. At this point it seems that something has to give - either use normal compression, or don't have access to tompe's MobiPerl tools, I guess.

Precisely.

I'm waiting to hear if documentation for the high compression algorithm exists. I suspect not. They state on their forums that they can't use standard zip compression because they need to be able to start uncompressing at any point in the file, and zip requires starting from the beginning, so they appear to have a complex algorithm that takes a long time to create a compressed file. (One poster talked about creating a dictionary where it took 24 hours to create the High Compression file.) My limited tests here indicate it decompresses about as quickly as Standard compression, so whatever they are doing is asymmetric.

And I can relate - I'm a pack-rat myself.

______
Dennis

[tompe]

Originally Posted by Jiiri

I think tompe is talking about files created with high compression not being able to be 'unpacked' or otherwise manipulated by his Mobiperl scripts. Since a lot of people (I am one) like to manipulate the metadata after we download a file, any file for distribution should be created using normal compression.

The meta data is not compressed so that is not a problem. What you loose is possibility to "explode" the file and fix something in the HTML code and then regenerate the mobi file.

But my objection is more philosophocal or ideological. I think it is bad of companies to use secret formats but i assume they feel they have to do it. But there is no reason to use a secret format for a file you distribute publicly. That is like just distributing binaries for programs and not distribute the source code which I also do not like.

But I assume if there is a future demand to unpack these files somebody will figure out the compression algorithm.

[tompe]

FBReader (at least my Debian Unstable version) cannot read files generated with high compression (-c2).

[cmbs]

I just don't feel obligated to provide you with something you can take apart, nor do I think anyone else is obligated to do so. There's nothing wrong with doing so, but at the same time, it's not obligatory.

Instead of complaining you can't take it apart, maybe you could have the attitude of being thankful for what someone did create and share.

[DMcCunney]

Originally Posted by tompe

The meta data is not compressed so that is not a problem. What you loose is possibility to "explode" the file and fix something in the HTML code and then regenerate the mobi file.

Sure. But depending upon the Mobi file, that might not be an option in any case. Will MobiPerl extract to HTML from a commercial title protected by DRM?

For titles uploaded here, I'd start by posting a bug report to the original creator. The folks creating Mobi content all seem to be quite receptive, and upload newer versions of titles as errors are pointed out and as they learn how to make better crafted Mobi titles.

But my objection is more philosophocal or ideological. I think it is bad of companies to use secret formats but i assume they feel they have to do it. But there is no reason to use a secret format for a file you distribute publicly. That is like just distributing binaries for programs and not distribute the source code which I also do not like.

I'm sympathetic to the objection, but for me, pragmatism trumps ideology. For instance, I prefer open source software when I can get something that meets my needs. I can't, always.

And a fair amount of stuff is proprietary, like it or not. For instance, you post MobiPerl files as RAR archives. While an assortment of tools can extract them, as far as I know, only the closed source and commercial RAR and WinRAR products can create them. (If there is an open source product that creates RAR archives, I'd be delighted to hear about it.)

Why not use something like 7zip, which is open source?

But I assume if there is a future demand to unpack these files somebody will figure out the compression algorithm.

Or MobiPocket will document it well enough to allow folks to decompress it with third-party tools.
______
Dennis

[DMcCunney]

Originally Posted by cmbs

I just don't feel obligated to provide you with something you can take apart, nor do I think anyone else is obligated to do so. There's nothing wrong with doing so, but at the same time, it's not obligatory.

He does have an objection I consider valid: the FBReader ebook viewer displays Mobi documents, but can't handle the high compression variant. FBReader runs on an assortment of things MobiReader doesn't: http://www.fbreader.org/downloads.php

While I prefer Plucker for my own use, Plucker is largely PalmOS specific. (There is a product called Vade Mecum that reads Plucker files on PocketPCs, and FBReader reads Plucker files, too.)

I'm experimenting with MobiPocket because it seems to be the most broadly supported format. If I'm going to create ebooks for others to read, I want to make them available to the widest possible audience.
______
Dennis

[tompe]

Originally Posted by DMcCunney

Sure. But depending upon the Mobi file, that might not be an option in any case. Will MobiPerl extract to HTML from a commercial title protected by DRM?

No, and for DRM:ed files there is no issue.

For titles uploaded here, I'd start by posting a bug report to the original creator. The folks creating Mobi content all seem to be quite receptive, and upload newer versions of titles as errors are pointed out and as they learn how to make better crafted Mobi titles.

You assume that the original creator is still around. What will happen in many cases is that files cannot be enhanced because the original creator is not around and somebody starts over again duplicating the work.

And a fair amount of stuff is proprietary, like it or not. For instance, you post MobiPerl files as RAR archives. While an assortment of tools can extract them, as far as I know, only the closed source and commercial RAR and WinRAR products can create them. (If there is an open source product that creates RAR archives, I'd be delighted to hear about it.)

I did not now that you could create zip files on a Linux machine until yesterday... I thought the the licence for ZIP files where more problematic or that the licence for programs to create ZIP files were worse. But I might have misunderstood things.

Or MobiPocket will document it well enough to allow folks to decompress it with third-party tools.

Does Mobipocket Creator allow you to decompress the files?

[Jiiri]

Originally Posted by cmbs

I just don't feel obligated to provide you with something you can take apart, nor do I think anyone else is obligated to do so. There's nothing wrong with doing so, but at the same time, it's not obligatory.

Instead of complaining you can't take it apart, maybe you could have the attitude of being thankful for what someone did create and share.

It is disingenuous to take what I or anyone else in this thread has written as 'complaining you can't take it apart.'

I do prefer to be able to re-format some things, and I said so in polite, friendly terms. This response is a little excessive for the discussion we're having, in my opinion, and is a little upsetting - I feel like the tone of the thread has been constructive. Responding in harsh tones is a great way to discourage new members like myself from posting their thoughts.

Jiiri

[DMcCunney]

Originally Posted by tompe

No, and for DRM:ed files there is no issue.

There is if you would like to fix something...

You assume that the original creator is still around. What will happen in many cases is that files cannot be enhanced because the original creator is not around and somebody starts over again duplicating the work.

So?

There are files posted to MobileRead now that could be considered duplications of effort. Look at some of the files HarryT has been releasing, where he has put substantial effort into proofing and correcting the source text to produce a more accurate version. It's probably faster and simpler to create a whole new release than to try to correct and existing one in such a case.

I did not now that you could create zip files on a Linux machine until yesterday... I thought the the licence for ZIP files where more problematic or that the licence for programs to create ZIP files were worse. But I might have misunderstood things.

The zip format was created by the late Phil Katz, who did business as PKWare. He started out making products that were improvements on the existing MS-DOS "ARC" archiver from a company called Seaware. Phil's PKArc and PKXArc provided better and much faster compression and decompression than the ARC equivalents. Seaware sued him for rights violations, as they had not released details of their archive format or permitted its use by other products.

Phil countered by creating PKZip, with a completely different archive format. He explicitly dedicated the Zip format to the public domain, so anyone could create software to create and manipulate Zip files, and many folks have done so. Open source products called zip and unzip have been available for Unix/Linux/BSD for years.

I'm startled you just discovered you could create zip files under Linux.

Does Mobipocket Creator allow you to decompress the files?

No. It allows you to compress them. It you have MobiPocket Creator, it's assumed you have the source texts, and you can do another build at will. There's no reason for it to decompress the files.
______
Dennis

[tompe]

Originally Posted by DMcCunney

There are files posted to MobileRead now that could be considered duplications of effort. Look at some of the files HarryT has been releasing, where he has put substantial effort into proofing and correcting the source text to produce a more accurate version. It's probably faster and simpler to create a whole new release than to try to correct and existing one in such a case.

Of course. But why increase the probability for it?

And if you are proofing a text it must be better to start with a version you have proofed three times already then start from scratch.

Phil countered by creating PKZip, with a completely different archive format. He explicitly dedicated the Zip format to the public domain, so anyone could create software to create and manipulate Zip files, and many folks have done so. Open source products called zip and unzip have been available for Unix/Linux/BSD for years.

When I installed a zip program in Windows it said something about not being free and I was unsure if I could use it to create files to distribute. But I might just have downloade the wrong version or misread something.

I'm startled you just discovered you could create zip files under Linux.

I am really a Unix and open source (or GNU) person and only uses Windows under protest... So I usually send tar files to people asking for zip files (rar-program can unpack them).

No. It allows you to compress them. It you have MobiPocket Creator, it's assumed you have the source texts, and you can do another build at will. There's no reason for it to decompress the files.

Of course there can be reasons. If you have lost your source files for example. So there are no non-third part programs that unpacks a MobiPocket file.