Error converting very large ePub

[dfabulich]

I have an ePub containing 380MB of pure text. It's 58 million words, no graphics, in 1806 chapter files. It's a computer-generated ePub; most of the pages are duplicates. (I'll explain in a postscript why I'm doing this ridiculous thing.)

Anyway, when I try to convert it to Mobi with Amazon kindlegen, it crashes with a Segmentation Fault, so I thought I'd try Calibre. I'm using the command-line tool, ebook-convert, and it seems to be going so well, until eventually it crashes, too, with this error:

Generating MOBI index for a book
Python function terminated unexpectedly: 'H' format requires 0 <= number <= 65535
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 154, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/ebooks/conversion/cli.py", line 393, in main
  File "site-packages/calibre/ebooks/conversion/plumber.py", line 1238, in run
  File "site-packages/calibre/ebooks/conversion/plugins/mobi_output.py", line 212, in convert
  File "site-packages/calibre/ebooks/conversion/plugins/mobi_output.py", line 249, in write_mobi
  File "site-packages/calibre/ebooks/mobi/writer2/main.py", line 63, in __call__
  File "site-packages/calibre/ebooks/mobi/writer2/main.py", line 77, in dump_stream
  File "site-packages/calibre/ebooks/mobi/writer2/main.py", line 239, in generate_record0
error: 'H' format requires 0 <= number <= 65535

I get it, I get it, it's too big. But what is too big, exactly? Is it the number of words? Number of chapter files? Is there a way I could predict when a book will be "too big" for Mobi and avoid those limitations?

P.S. OK, I promised I'd explain what I'm doing. What I'm doing is building a "choose-a-path" gamebook. In traditional "choose-a-path" books, like "Choose Your Own Adventure," every page is unique, and the experience of reading the book once is quite short. In other gamebooks, like the "Fighting Fantasy" gamebooks, choices you make early on in the story can affect what happens later in the story; they ask you to take notes with pencil and paper. After exploring the book, you might find a door, and the text would say, "If you've found the key, you can unlock the door and turn to page 127."

In the interactive-fiction community we call these two types of gamebooks "stateless" or "stateful," where the idea is that what you write on the paper is the "mutable state" of the game world.

It turns out that if you don't have very much mutable state, you can generate an enormous stateless gamebook from a stateful gamebook. The input source for this ePub is only 30,000 words, 208KB of text, but from that we generate an ePub that tracks your choices as you play, so you don't have to use pencil and paper. The result is 58 million words, but it also compresses surprisingly well, 19MB.

Anyway! I'd like to take this monster and convert it into a Mobi file; if that's not possible, I'll have to reduce the amount of tracked state to make it fit the constraints of the platform.

But to do that, I'd need to know what my constraints are! How big can I go, in which dimensions?

[kovidgoyal]

The total text length is to large for MOBI. MOBI files are internall Palm databases. The text is split into multiple records. Each record has a fixed maximum length and there are at most ~ 65K text records.

You might get somewhere if you try kindlegen with huffman compression. Though I'd be fairly surprised that even if you succeeded in generating the MOBI ti would actually work on kindle devices.

[kovidgoyal]

IIRC text record size is 4096 bytes

[mbovenka]

I think you're going past the maximum of 64K PDB Records in a MOBI file: https://wiki.mobileread.com/wiki/MOBI#PalmDOC_Header.

Looking at that, an uncompressed MOBI file can't be larger than 4K x 64K = 256M. I'm not sure what Calibre does; if it uses Huffman compression, it should fit, but perhaps it doesn't.

Edit: Ninja-ed :-)

[dfabulich]

Thanks so much, folks! Another quick question. When I'm running ebook-convert, it seems to take a huge amount of time "Removing fake margins..." Is there a way to skip that step?

Relatedly, it appears that it's first converting the ePub to HTML and then converting the HTML into Mobi. But the ePub is already a big zip of HTML, isn't it? Would it save time if I generated HTML directly? (If so, would I generate a single enormous 200MB HTML file, or would a collection of HTML files work?)