Reverse-engineering the .IMP format

[nrapallo]

A primer on the .IMP specification has never been published, but a very detailed explanation of the .IMP file format can be found here. It was reverse-engineered by Jeffrey Kraus-yao back in 2002. Jeffrey indicated that how he reversed engineered the .imp format was by building, with eBook Publisher, .oeb test ebooks. Then he would drop that .oeb onto a desktop shortcut of the imp viewer.exe. Then while the viewer was still running, he would examine his temp folder and noticed one of four .RES folders was complete. He wrote down the changed bits and then repeatedly made small changes to the .oeb ebook. He started this even though he didn't own the REB1200 yet.

It was quite the accomplishment! Now please realize that back then only the REB1200 used the .IMP format as there weren't a lot of GEB1150's (predecessor to the EBW1150) in 2002/3. Oh, how the tables have turned as now for every REB1200 in use there are tens or hundreds more EBW1150's!

Jeffrey's website is a great start to understanding the .IMP file format, but lacks such basic information as:

• clarification about all image types allowed to be stored in a .imp ebook;
• byte ordering (LittleEndian vs. BigEndian) differences between the EBW1150 and REB1200 .imp types;
• how to identify the .imp type i.e. for EBW1150/GEB1150 or for REB1200/GEB2150;
• fact that DATA.FRK (compressed or uncompressed) is the same for the EBW1150 AND REB1200;
• fact that images are stored in their original resolution and in color even though they may be reduced to fit the device screen and color depth;
• some more "unknown" .RES filetypes like 'Devm', 'Form', etc.

You may have even noticed that the EBW1150 .imp is slightly bigger (filesize-wise) than the same REB1200 .imp when no images are present. That is because the EBW1150 .imp uses two additional (irrelevant/unused) bytes for most records and when multiplied by thousand of records results in a larger file! I think a different "programming team" came out with the EBW1150 .imp file format as most .RES filetype records are reversed byte wise (i.e. BE vs LE). It makes this .IMP reverse-engineering unnecessarily more difficult!

I would like to herein build a knowledge-base for the "definitive" understanding of the .IMP file format. As others have already expressed to me their own foray into .imp "nuts & bolts" investigations, I propose to start off this knowledge-base with my preliminary findings written as a Perl script. That script is imp_dump.pl (along with it's required support files) and can be used to exploded any un-encrypted .imp ebook into it's (decompressed) text and images components.

Now, take note, that I said text and images NOT .html and images.

The original html is not stored in the .imp file. Only the basic components are, like a record that tells you where all the font/styles changes are located in the file, another record indicates where to end the line so that it doesn't spill over the screen size of that .imp and other records that stores the images, hyperlinks used, etc. Basically all the building blocks are there (scattered) and we require those components to be re-assembled somehow into a .html!

BTW, release v4.0 of EBook-Tools should have basic .imp support for .html generation with image linking, but will initially lack table/hyperlink/styles support. Those are planned for future releases.

I plan to collect postings from this thread and compile a wiki page with the relevant parts of the .IMP file format specification as reverse-engineered by ALL of us!

Below are all the .RES filetypes that exist (thus far) and volunteers can pick the un-documented .RES filetypes on a "first come, first serve" basis.

.IMP file comprises these groups of .RES filetypes:
	text
	page_line
	page_header_footer
	links
	misc_info
	formatting
	tables
	images
	markups
	form_data

where:

text:
	'!!cm'
	'!!ky'
	DATA.FRK - decompressor written in Perl, C and soon to be C#.

page_line:
	'BPgz'
	'BPgZ'
	'ImRn' - written in Perl (see imp_dump.pl)
	'Pcz0' - written in Perl (see imp_dump.pl)
	'PcZ0' - written in Perl (see imp_dump.pl)
	'Pcz1' - written in Perl (see imp_dump.pl)
	'PcZ1' - written in Perl (see imp_dump.pl)

page_header_footer:
	'HfPz'
	'HfPZ'

links:
	'AncT' - written in Perl (see imp_dump.pl)
	'AnTg'
	'Lnks'
	'eLnk'

misc_info:
	'Batr'
	'Binf'
	'BGcl' - written in Perl (see imp_dump.pl)
	'BPos'
	'Clos'
	'Devm' - written in Perl (see imp_dump.pl)
	'Dict'
	'FRgs'
	'Glos'
	'MASK'
	'Mrgn' - written in Perl (see imp_dump.pl)
	'Hyp2'
	'Hyph'
	'Offs'
	'pInf' - written in Perl (see imp_dump.pl)
	'Pc31'
	'PPic' - written in Perl (see imp_dump.pl)
	'SKtb'
	'SMnu'
	'stbd'
	'!!sw' - written in Perl (see imp_dump.pl)

formatting:
	'ESts' - written in Perl (see imp_dump.pl)
	'HRle'
	'Styl'
	'StRn' - written in Perl (see imp_dump.pl)
	'StR#'
	'StR2'

tables:
	'Tabl'
	'TCel'
	'TRow'

images:
	'GIF ' - written in Perl (see imp_dump.pl)
	'JPEG' - written in Perl (see imp_dump.pl)
	'PIC2' - written in Perl (see imp_dump.pl)
	'PICT' - written in Perl (see imp_dump.pl)
	'PNG ' - written in Perl (see imp_dump.pl)

markups:
	'MRPs'
	'Ano2'
	'Hlts'
	'BTok'
	'BMks'

form_data:
	'TGNt'
	'Form'
	'FItm'
	'FIDt'
	'FrDt'

I'm calling out to any and all interested in this detailed examination of the .IMP file format! We need your findings summarized as either (1) Perl code or (2) offset / length with short narrative like Jeffrey's website uses. If you're so inclined, consider updating my imp_dump.pl to incorporate your findings and re-upload it here.

What you'll need is a test .imp, a good binary/hex editor (I use XVi32 Edit) and a lot of elbow grease and desire. Post here what you find out and I'll update the "un-documented" list above to reflect that!

Thanks in advance!

p.s. after unzipping the attachment, just place any and all your .imp files in the folder therein called 'place imp file here' and execute the 'extract imp files.bat'. Look at the generated file 'imp_dump.output.txt' for the parsing output info for all the .imp files placed in that folder. Then, look in that folder to see a directory for each .imp that will contain the compressed & decompressed text and any images. Have fun HEX-exploring!

EDIT: 06Jan2009: added a compiled windows executable to a separate .zip

[nrapallo]

Please refer to my original posting of deimp.exe here for some background info.

One of the component records in the .RES directory when the .IMP file is exploded with unimp.exe is the DATA.FRK file. It contains the basic text used in the ebook and is the same for both the Color VGA (REB 1200) & Grayscale Half-VGA (EBW 1150) .IMP files. This DATA.FRK file is decompressed by deimp.exe if it was originally (LZSS) compressed, when created, along with control characters (see below) being substituted/expanded.

DATA.FRK File

Element text is extracted and placed in this file. Elements tags are replaced with control characters. This file can be compressed and encrypted with compression occuring before encryption. This file is compressed when the element <meta name="x-SBP-compress" content="on"/> is included in the <x-metadata> element of the package file. The compression algorithm used is LZSS. This file is encrypted when the element <meta name="x-SBP-encrypt" content="on"/> is included in the <x-metadata> element of the package file. The encryption algorithm used is DES. The 8 byte encryption key is in the SoftBook Edition Encryption Key File (.key) at offset 0x0C.

Characters less than 0x20 are removed expect for line break which is replaced with 0x20. Mutliple 0x20 characters are replaced with a single 0x20.

Control characters

0x0A end of document, forced page break 
0x0B start of element except < span >
0x0D line break element < br / >
0x0E start of table element < table > 
0x0F image element < img / > 
0x13 end of table cell < /td > tag 
0x14 horizontal rule element < hr / > 
0x15 before and after page header content 
0x16 before and after page footer content

As previously stated, my deimp.exe program used as it's base the lzss-0.6 code by Michael Dipperstein (http://michael.dipperstein.com/lzss), with tweaks by me to get it to decode the .imp text. I added the ability to insert/substitute some characters that are not part of the lzss decompression so that the resulting .imp text looked better. Just remove those and then after decompression, you can substitute them back.

In addition to those control characters above, characters to "substitute/convert" would be:

        HEX => Should be (actual char)
        0x8E => "é"  (i.e. "é"),
        0xA0 => " ",   (i.e. " "),
        0xA5 => "•",     (i.e. "•"),
        0xA8 => "®",     (i.e. "®"),
        0xA9 => "©",   (i.e. "©"),
        0xAA => "™",  (i.e. "™"),
        0xAE => "Æ",   (i.e. "Æ"),
        0xC7 => "«",  (i.e. "«"),
        0xC8 => "»",  (i.e. "»"),
        0xC9 => "…",   (i.e. "…"),
        0xD0 => "–",  (i.e. "–"),
        0xD1 => "—", (i.e. "—"),
        0xD2 => "“",    (i.e. "“"),
        0xD3 => "”",   (i.e. "”"),
        0xD4 => "‘",    (i.e. "‘"),
        0xD5 => "’",   (i.e. "’"),
        0xE1 => "·",  (i.e. "·"),

I attach the source code to my deimp.exe (and original lzss-0.6) below for your use and further study. Please excuse the coding hacks as this was a work-in-progress until I "nailed" the decompression algorithm. It didn't lend itself to good programming style.

p.s. as an exercise, would anyone want to try tweaking this code to allow the LZSS (re-)compression of text for use as the DATA.FRK in the .imp?

[mscott161]

Hello All,

With Nick's help, I was able to put together a simple de-imp program to pull the text out of the imp file. I appologize for the lack of documentation and plan to add that to the code soon. Please feel free to submit any changes to the code. I am working on a GUI front end so you can select a single file or folder and the program will de-imp the files to text and hopefully with some help to the file format of lrf (Sony).

Thanks to all.
-Michael

[nrapallo]

Originally Posted by mscott161

I am working on a GUI front end so you can select a single file or folder and the program will de-imp the files to text and hopefully with some help to the file format of lrf (Sony).

GUI? GUI! That would be a welcomed addition to the current dos tools!

Nice to see you are already adding to our knowledge-base!

Thanks again for sharing this!

[mscott161]

Nick,

I agree with you on the lack of something to look at besides the console. So here is a GUI version of my previous ConvertIMP.

I know it is not much, but it is something to start with.

I will try to put in aknowledgements in the next update to the program but till then I would like to thank Nick and Michael Dipperstein for their libraries that helped me produce the program.

-Michael

New addition to v.1.0.1 - Image Viewing

New addition to v.1.0.2 - Image Viewing (including PNG, GIF, JPEG)

New addition to v.1.1.0 - Editting Book Properties. --- I am removing this pending furthure testing. Sorry for the delay.

[nrapallo]

Originally Posted by mscott161

So here is a GUI version of my previous ConvertIMP.

Looks very promising!!!

So when will we be able to edit the metadata? v1.1?

[mscott161]

What would you like to edit?

--Michael

[nrapallo]

Originally Posted by mscott161

What would you like to edit?

--Michael

Where do I begin (as shouts can be heard from around the world from .imp users! ):

Per the .imp specs:
Edit - Book properties start at offset 0x30
Yes* - ID: null terminated C string
YES - Bookshelf Category: null terminated C string
N/A - Subcategory: null terminated C string, not displayed on REB1200
YES - Title: null terminated C string
N/A - Last name: null terminated C string
N/A - Middle name: null terminated C string
YES - First name: null terminated C string

Note * = there is a way to auto-generate this ID.; N/A = not allowed; YES = allow edits

Afterwards, the length of Book Properties (including 7 null's) needs to be updated so that BytesRemainingInHeader is set to length of Book Properties + 24!

Also, it would be nice if the Name of .RES directory could be changed to the .imp filename (minus .ext) or even auto-generated to be 'Author-Title' as you used for the (decompressed) text filename. The DictionaryLength (length of directory name) in the 48 byte header would have to be updated then as well. (p.s. you called it Dictionary, but I think you meant Directory...)

That's it for now...

Check please!

[DaleDe]

Originally Posted by nrapallo

Where do I begin (as shouts can be heard from around the world from .imp users! ):

Per the .imp specs:
Edit - Book properties start at offset 0x30
Yes* - ID: null terminated C string
YES - Bookshelf Category: null terminated C string
N/A - Subcategory: null terminated C string, not displayed on REB1200
YES - Title: null terminated C string
N/A - Last name: null terminated C string
N/A - Middle name: null terminated C string
YES - First name: null terminated C string

Note * = there is a way to auto-generate this ID.; N/A = not allowed; YES = allow edits

Afterwards, the length of Book Properties (including 7 null's) needs to be updated so that BytesRemainingInHeader is set to length of Book Properties + 24!

Also, it would be nice if the Name of .RES directory could be changed to the .imp filename (minus .ext) or even auto-generated to be 'Author-Title' as you used for the (decompressed) text filename. The DictionaryLength (length of directory name) in the 48 byte header would have to be updated then as well. (p.s. you called it Dictionary, but I think you meant Directory...)

That's it for now...

Check please!

I think the big things a user would like to correct include: category!!!, Author, Title.

Dale

[mscott161]

I have updated the ConvertIMPGUI program to include viewing of the images in the IMP file. Please look at the attachments in a previous message for the update.

I will look into the category, author, and title changes.

--Michael

[nrapallo]

Originally Posted by DaleDe

I think the big things a user would like to correct include: category!!!, Author, Title.

Dale

And just to add that eBookwise/GEB Librarian do an excellent job allowing users to edit and setup their own categories to use when editing the category.

Basically, the users choose the category (when editing the metadata) from a drop-box list that includes predefined categories plus user previoulsy used/defined ones.

They do say that imitation is the greatest form of flattery...

Just food for thought!

[mscott161]

Sorry about the image viewing before I only included PNG. I have updated the code in the previous message (see new attachment to that message for v.1.0.2) which now include (PNG, GIF, and JPEG).

--Michael

[mscott161]

Nick and Everyone,

I have put in book property editing in the release v.1.1.0 Hope you enjoy. I am all ears if something does not work correctly or ideas on GUI arrangement. If you have a specific idea on GUI send a screen shot of what you want it to look like and I will take a look.

I updated the ConvertIMPGUI Post above with the new version.

Happy Holidays
--Michael

[nrapallo]

Originally Posted by mscott161

Nick and Everyone,

I have put in book property editing in the release v.1.1.0 Hope you enjoy. I am all ears if something does not work correctly or ideas on GUI arrangement. If you have a specific idea on GUI send a screen shot of what you want it to look like and I will take a look.

I updated the ConvertIMPGUI Post above with the new version.

Happy Holidays
--Michael

As we are still testing the waters with .imp editing, please try using ConvertIMP on backup copies only until you verify editing works for you. I've had a few issues already that I notified Michael about.

Please proceed with caution with these early releases!

[mscott161]

I have started a new thread with Nicks help. I will post all release and changes to it.

-- Michael