DR1000 First impressions from a late newcomer

[fekhner]

Hi everybody,

I have been hanging around here for some time trying to buy the right e-reader for me and would like to share my experience so far. Most probably, it will remind you of what you were experiencing a couple of years ago.

First of all, thanks so much for all the help and work already done, amazing, this forum is unbelievable!

My reading needs are mostly professional: pdf papers and books, djvu books, organized on an apple laptop in bibdesk for the papers and calibre for the books; bibdesk because it allows you to organize the papers and interacts well with aquamacs, calibre in the beginning only in order to organize books.

From my hardware I was more or less destined to go for an ipad, but somehow the commercial side of it and the impression that you do not really get full access to the device prevented me from doing so. In the end I got a DR1000 from an auction on ebay germany, it is of course a gamble, an "old" device from a company that no longer exists with a prize similar to an ipad ...in any case, what clinched it for me was the screen of course, but also the djvu plugin, thanks so much luite!

Then there is the great post of Grimulkan on the hardware, so I ordered a Kingston card and realized that I do have unused motorola razr chargers both for the car, wall and additional usb ones.

The device came last week with FW2.0-rc3 already on it. So I put Xournal, thanks Mackx! and also for the patch that I played with a little, I guess it is the only way to re-shuffle the files on the device, just doing it from the computer won't work ?

I also put the libs from Iñigo and tried Fbreader and also fb2 from yesterdays posts, but both somehow freeze when I try to open a chm file ... I guess I still have some reading to do, thanks anyway and please do not forget the DR1000 when you develop for the DR800 .

Finally calibre does not seem to be able to export djvu or chm to the DR1000, the plugin does not seem to be aware of the great work all of you have done. I have opened a ticket for djvu with calibre, but maybe some of the more qualified people here should do that because they will know better what other types of books also work.

Any other suggestions or tips for programs to put, the optimal configuraton etc is of course most welcome! Thanks to all earlier hardcore developers Adam, Gertjan and all those I forget ...

The next post will be at most two lines long, it is a promise!

[Mackx]

Hi fekhner,

Thanks for posting your first impression.

Originally Posted by fekhner

I also put the libs from Iñigo and tried Fbreader and also fb2 from yesterdays posts, but both somehow freeze when I try to open a chm file ... I guess I still have some reading to do, thanks anyway and please do not forget the DR1000 when you develop for the DR800 .

There is a DR1000 specific FBReader version, but it is well hidden... See this thread message#86 and #89. I use this vesion to read chm-files, so if it still does not work, it might be related to the chm-file. Support for chm in FBReader is not optimal.

[luite]

Hi fekhner, good to see that you like my plugin. Unfortunately I can't use it anymore, since my own DR1000S is broken. I will however update it if the plugin interfaces change.

I bought an iPad to replace the broken DR1000, and I have to agree with your observations. The screen is less readable in many situations, most applications are closed source, and not free (as in beer). While I don't mind paying a few euros for a good application (I bought some), I hate not being able to improve them. Most PDF viewers use Apples PDF renderer, which has some severe limitations (see below), and it's not possible to change this without the source code.

Anyway, I've been working on a DJVU to PDF conversion program that tries to retain the good compression of the DJVU file, without much quality loss. It also converts the text and table of contents (if present).

Making a small PDF file from a DJVU file is a challenge, especially for Apples PDF renderer, and I haven't found any program (free or commercial) that does a good job at this.

First some basic information about DJVU: Every DJVU page can contain a few layers:
- background, typically a medium-resolution color image, contains the paper color and color images.
- mask, contains the data where the foreground layer is visible over the background, typically a high-resolution bitonal (1 bit, black/white) JB2 image containing the shapes of the letters and line drawings
- foreground layer, contains the color of the letters, typically low resolution

See the attached image for an example of how djvu segmentation works.

The key to retaining good compression in the conversion is converting all these layers separately. PDF supports image masks and multiple images per page, so we should be good, right? Unfortunately things are not so good for Apple users, since their PDF renderer has two major limitations:
- It does not support JPEG2000 images (so we have to fall back to the inferior JPEG format for the foreground and background layers)
- Applying a high resolution mask (the mask layer) to a low resolution image (the foreground layer), results in a low resolution image, with in many cases unreadable text.

Fortunately, I found a way around the second issue: If you apply the image mask to a softmask loaded in the graphics state, instead of directly applying it to the image, you can get a high resolution result. There are still some issues with this, the colors in Adobe Reader are slightly different from the original colors. I don't know what causes this, but I want to resolve it before I release the tool and the source code. Perhaps it's some color space or color management issue.

So what my conversion program does for each page is the following:
- convert background layer, if present, from IW44 to JPEG (remove background layer if it's nearly white)
- convert foreground layer from IW44 to JPEG (remove it if it's nearly all black)
- convert mask layer to a high compression JBIG2 image (with the JBIG2 symbol table shared between multiple pages for better compression)
- render the foreground layer over the background layer with the mask applied
- render the hidden text from the OCR layer

Anyway, results, for example this book:
http://www.archive.org/details/happy...ndot00wilduoft
- DJVU file: 6.3MB
- PDF file: 42MB
(I count a MB as 1 million bytes)

The PDF file contains many JPEG2000 images, so all programs that uses Apples PDF renderer on the iPad cannot render them, resulting in many blank pages (this includes GoodReader, iAnnotate PDF)

Output from my program, converted from the DJVU file:
http://94.124.88.78/~luite/happyprin...00wilduoft.pdf
- PDF 10MB

It's not quite as small as the original DJVU file, mainly due to the worse JPEG compression (and I could optimize the rendering of the hidden text layer a bit more), but it is fully iPad compatible (Note: it doesn't set the DPI of the page properly yet, so viewing it at 100% will probably result in huge pages)

Once the color conversion issue is resolved, I'll post the source code here (or maybe in the Apple Devices or PDF forum). I'm also trying to improve the compression and pdf rendering speed by identifying more cases when the foreground layer can be removed (by using multiple JBIG2 images, each for a single color).

I hope other djvu users will find this useful

[Iñigo]

Originally Posted by fekhner

Hi everybody,

I have been hanging around here for some time trying to buy the right e-reader for me and would like to share my experience so far. Most probably, it will remind you of what you were experiencing a couple of years ago.

First of all, thanks so much for all the help and work already done, amazing, this forum is unbelievable!

My reading needs are mostly professional: pdf papers and books, djvu books, organized on an apple laptop in bibdesk for the papers and calibre for the books; bibdesk because it allows you to organize the papers and interacts well with aquamacs, calibre in the beginning only in order to organize books.

From my hardware I was more or less destined to go for an ipad, but somehow the commercial side of it and the impression that you do not really get full access to the device prevented me from doing so. In the end I got a DR1000 from an auction on ebay germany, it is of course a gamble, an "old" device from a company that no longer exists with a prize similar to an ipad ...in any case, what clinched it for me was the screen of course, but also the djvu plugin, thanks so much luite!

Then there is the great post of Grimulkan on the hardware, so I ordered a Kingston card and realized that I do have unused motorola razr chargers both for the car, wall and additional usb ones.

The device came last week with FW2.0-rc3 already on it. So I put Xournal, thanks Mackx! and also for the patch that I played with a little, I guess it is the only way to re-shuffle the files on the device, just doing it from the computer won't work ?

I also put the libs from Iñigo and tried Fbreader and also fb2 from yesterdays posts, but both somehow freeze when I try to open a chm file ... I guess I still have some reading to do, thanks anyway and please do not forget the DR1000 when you develop for the DR800 .

Finally calibre does not seem to be able to export djvu or chm to the DR1000, the plugin does not seem to be aware of the great work all of you have done. I have opened a ticket for djvu with calibre, but maybe some of the more qualified people here should do that because they will know better what other types of books also work.

Any other suggestions or tips for programs to put, the optimal configuraton etc is of course most welcome! Thanks to all earlier hardcore developers Adam, Gertjan and all those I forget ...

The next post will be at most two lines long, it is a promise!

Welcome Fekhner to the fascinating world of DR devices ;-)

FBreader behaviour with chm files is far from perfect in DR devices.
If I remember correctly, chm is a format which compress html files.
It looks like FBReader uses too much memory to uncompress it, not a real problem on a desktop computer, but it is on the DR that doesn't have much RAM.

The best advice is to avoid chm files. There are some tools out there that convert them to .pdf files, but they didn't work perfectly on my tests.

[fekhner]

Originally Posted by Mackx

Hi fekhner,

Thanks for posting your first impression.


There is a DR1000 specific FBReader version, but it is well hidden... See this thread message#86 and #89. I use this vesion to read chm-files, so if it still does not work, it might be related to the chm-file. Support for chm in FBReader is not optimal.

Thanks, FBreader does indeed work perfectly now. I will follow yours and Iñigo's advice and stay clear of chm files, in any case I do not really have to use them for the moment.

[fekhner]

Originally Posted by luite

Hi fekhner, good to see that you like my plugin. Unfortunately I can't use it anymore, since my own DR1000S is broken. I will however update it if the plugin interfaces change.

I bought an iPad to replace the broken DR1000, and I have to agree with your observations. The screen is less readable in many situations, most applications are closed source, and not free (as in beer). While I don't mind paying a few euros for a good application (I bought some), I hate not being able to improve them. Most PDF viewers use Apples PDF renderer, which has some severe limitations (see below), and it's not possible to change this without the source code.

Anyway, I've been working on a DJVU to PDF conversion program that tries to retain the good compression of the DJVU file, without much quality loss. It also converts the text and table of contents (if present).

....


I hope other djvu users will find this useful

In fact I read that you do not any longer have a DR1000s and was worried, great that you still follow what is going on!

I also believe that there is a real need for a good conversion tool, for instance most of the old mathematics and physics books are in djvu. On the Dr1000 at least, there seems to be no need for conversion as I find the result produced by the djvu plugin extremely readable, in fact better than what I expected.

I wanted to play with the stanza djvu viewer on the ipad, but never got my hands on it. How does it fare?

[dima_tr]

sorry, repeated your thought

[luite]

Originally Posted by fekhner

In fact I read that you do not any longer have a DR1000s and was worried, great that you still follow what is going on!
I wanted to play with the stanza djvu viewer on the ipad, but never got my hands on it. How does it fare?

It's quite useless actually, since it doesn't display djvu files directly, but converts them to pdf or something similar. This takes a long time, and results in huge files. There are a few other programs that can read djvu files, but none of them comes close to the pdf viewers.

[fekhner]

Originally Posted by luite

Anyway, I've been working on a DJVU to PDF conversion program that tries to retain the good compression of the DJVU file, without much quality loss. It also converts the text and table of contents (if present).
....

Once the color conversion issue is resolved, I'll post the source code here (or maybe in the Apple Devices or PDF forum). I'm also trying to improve the compression and pdf rendering speed by identifying more cases when the foreground layer can be removed (by using multiple JBIG2 images, each for a single color).

Hi luite,

Any update on your conversion program ? Even if its is not perfect, for instance color is not an issue for me at all, I for one would be very interested.

Cheers, F!

[luite]

hi fekhner,

sorry I abandoned that project since I wasn't satisfied with the rendering of the output on the iPad. The iPad (1) has a low resolution display, much lower than the DR1000, and the scaled bitmaps looked rather blurry (giving me a headache after a while). I have now converted all my scanned books with ClearScan OCR (Adobe Acrobat 9 and higher), and they look much sharper than with the direct DjVu conversion.

The new iPad with its high res 2048x1536 display might solve the blurriness issue though. I might get one, so maybe it's time to revisit this conversion utility then