PDFRead - reading PDFs on the 1100/1150/1200 eBook Readers

[alex_d]

Ha, I love it when someone asks "what features can i add?" And I love it even more that I know that you're the kind of guy to actually implement them.

Here are some things that I know people want given my experience gathering feedback for pdfrasterfarian:

Working with images as input (something I never got around to doing myself):
Lots of people have folders of images of scanned books or comics they'd like to collate. Part of this feature is the ability to process double-page scans. Ie, you first split the scan down the middle (down a user-adjustable line) and then again if you're converting into landscape mode. This was one of the most popular feature requests I received.

Some minor stuff:
When a user selects a file, the format (pdf/djvu) can be selected automatically based on the extension. Also, when a user selects a profile, the file extension (eg .lrf) MUST be added automatically. (Otherwise the device won't open the file and the user won't know why.)

Some more minor stuff:
Try to add explanations for your options, particularly on the "customize" page. Even I am not sure what some of the things mean, and you're really robbing your users by taking away configurability (either completely or just in practice). A good way to add explanations is a box at the bottom whose contents change depending on where the mouse points.

Some more minor stuff, continued:
When you give people configuration options, and even if you add good explanations, there should be a way to preview the result.

[ashkulz]

Ha, I love it when someone asks "what features can i add?" And I love it even more that I know that you're the kind of guy to actually implement them.

Bring 'em on!

Working with images as input (something I never got around to doing myself):
Lots of people have folders of images of scanned books or comics they'd like to collate. Part of this feature is the ability to process double-page scans. Ie, you first split the scan down the middle (down a user-adjustable line) and then again if you're converting into landscape mode. This was one of the most popular feature requests I received.

Hmm, there's already a fantastic (but a tad intimidating) tool called unpaper which does all of that and more. As it has quite a few options to tweak the way it works, I'll have to see how to integrate it into the current pipeline (don't want to add too many options to PDFRead).

A quick and easy-to-implement way would be to write a wrapper GUI for unpaper, combine the unpaper output to a multi-page TIFF and then add TIFF support to PDFRead. That would solve most people's needs, and yet keep the complex unpaper configuration out of PDFRead.

Some minor stuff:
When a user selects a file, the format (pdf/djvu) can be selected automatically based on the extension. Also, when a user selects a profile, the file extension (eg .lrf) MUST be added automatically. (Otherwise the device won't open the file and the user won't know why.)

That's going to be a bit tough, as the Windows GUI isn't really a GUI. I've actually hijacked an installer (based on NSIS) to provide GUI-like features. Main reason: small code (~60kB) which works on all Windows versions with zero dependencies. I don't really want to get into the GUI thing for now, I want to focus on features (If someone does write a GUI, I'd be happy to include it -- all the current GUI does is build a command line and then execute it).

Some more minor stuff:
Try to add explanations for your options, particularly on the "customize" page. Even I am not sure what some of the things mean, and you're really robbing your users by taking away configurability (either completely or just in practice). A good way to add explanations is a box at the bottom whose contents change depending on where the mouse points.

You're right, some of the options will not make sense unless you know what is actually happening during all the various stages. I'll make a document about it and post it on the website sometime tomorrow.

Some more minor stuff, continued:
When you give people configuration options, and even if you add good explanations, there should be a way to preview the result.

The best way I can think of is to allow a page spec for conversion and convert for those pages only. ie. that way, you can convert a single page and see how it looks finally. Don't know how it will integrate with the windows GUI, though.

[alex_d]

"(don't want to add too many options to PDFRead)"

aww.... come on. Don't let your windows software get bogged down by the shortcomings of the command line. (I know this is the reason because you've mentioned it elsewhere.) The whole point of a GUI is that it allows you to effectively support and present many more features. If you don't want to add them to the linux version, that's fine.

"That's going to be a bit tough, as the Windows GUI isn't really a GUI. I've actually hijacked an installer (based on NSIS) to provide GUI-like features. Main reason: small code (~60kB) which works on all Windows versions with zero dependencies."

Oh.

"I don't really want to get into the GUI thing for now, I want to focus on features"

It's kind of hard to separate the two, seeing as it's hard to add features to the command line (and you're already claiming to have hit the ceiling).

I know the real reason you don't want to do a gui. It's because you're unfamiliar with how to do it and it does take quite a bit of work. (I know, because that's the exact same reason pdfrasterfarian stayed text-based.)

"(If someone does write a GUI, I'd be happy to include it -- all the current GUI does is build a command line and then execute it)."

Yeah, we need that person.

"You're right, some of the options will not make sense unless you know what is actually happening during all the various stages. I'll make a document about it and post it on the website sometime tomorrow."

Eh... a document isn't really the right solution. No one reads those things. A program should be its own document. I understand that your installer approach presents big limitations, but on the other hand it should have strong facilities to create a "wizard" to ask questions next to explanations similar to pdfrasterfarian.

"The best way I can think of is to allow a page spec for conversion and convert for those pages only. ie. that way, you can convert a single page and see how it looks finally. Don't know how it will integrate with the windows GUI, though."

It shouldn't be a problem if you can just call a command in the middle rather than the end.

[ashkulz]

aww.... come on. Don't let your windows software get bogged down by the shortcomings of the command line. (I know this is the reason because you've mentioned it elsewhere.) The whole point of a GUI is that it allows you to effectively support and present many more features. If you don't want to add them to the linux version, that's fine.

That's a bit out of context -- I don't want to add too many options unrelated to PDFRead (as the ones related to unpaper would be). In fact, I'd say that the command line has a better UI than any GUI, but that's something that everyone has [highly divergent] opinions on

It's kind of hard to separate the two, seeing as it's hard to add features to the command line (and you're already claiming to have hit the ceiling).

Uhm .. not really. I can (and will) add many more features related to the core functionality of PDFRead. I've not reached any kind of ceiling, it's just that I want to keep the current code lean and mean -- it's much easier to maintain stuff that way. I'd like to do a mini-rewrite (like I mentioned in the other thread) but that's not really necessary at all (I'm just very finicky when it comes to code quality).

I know the real reason you don't want to do a gui. It's because you're unfamiliar with how to do it and it does take quite a bit of work. (I know, because that's the exact same reason pdfrasterfarian stayed text-based.)

Now that's right on the mark I don't want to take the effort of maintaining a GUI, when it is much simpler to maintain a CUI. I have written GUIs (although now I do mostly web-based at work), but it takes much more code and effort. As a comparison -- all of PDFRead is ~620 lines, while the GUI code + configuration is ~518 lines (and that's very short, mind you).

Eh... a document isn't really the right solution. No one reads those things. A program should be its own document. I understand that your installer approach presents big limitations, but on the other hand it should have strong facilities to create a "wizard" to ask questions next to explanations similar to pdfrasterfarian.

Point taken, but some of the options will require you to understand the process and how that option fits in it. And yes, the installer approach does has its limitations -- but I got it up in less than a day, and it's good enough for everyday tasks. The folks who want to customize to the Nth degree can always use the command line...

It shouldn't be a problem if you can just call a command in the middle rather than the end.

I'm not sure I get what you mean.

[alex_d]

nope, command lines are worse because a) you're gonna hate typing even a measly dozen options and b) you're gonna have to keep switching to the man page just to remember/understand what to type.

It infuriates me how the *nix crowd doesn't understand the whole point of guis. They think it's just about pretty buttons and using the mouse. No, it's a fundamentally better way to present options and features.

It's also DEFINATELY not about mouse-vs-keyboard. (Well, on *nix it might be, but that's because the guis there are programmed so poorly.) A good gui will let you do everything from the keyboard. (E.g., ever notice those underlined letters?) You can use VS.net from the keyboard same as you can vi, but VS will expose 100x the features and the fraction of features you'll actually know how to use will also be far greater.

Of course none of that changes the fact that neither I nor you know how to do a good gui, but at least I don't try to esteem the limitations of my medium. The *nix crowd always does that, and it saddens me that *nix has hardly evolved in 30 years. 30 years!

On the other hand, though, I must say the command-line is very very useful when you're tying together various programs to do something new. Obviously that's the only way pdfrasterfarian was constructued, and I owe much gratitude. But if that's the rightful use of the command-line, then I think no one should have qualms with cramming in hundreds of options and parameters and a long manual that no human should ever have to use directly in the day-to-day.

This was an issue of discussion in the backend thread, and it surprised me when you said "no, the backend shouldn't have so many command-line options."

[ashkulz]

nope, command lines are worse because a) you're gonna hate typing even a measly dozen options and b) you're gonna have to keep switching to the man page just to remember/understand what to type.

It infuriates me how the *nix crowd doesn't understand the whole point of guis. They think it's just about pretty buttons and using the mouse. No, it's a fundamentally better way to present options and features.

It's also DEFINATELY not about mouse-vs-keyboard. (Well, on *nix it might be, but that's because the guis there are programmed so poorly.) A good gui will let you do everything from the keyboard. (E.g., ever notice those underlined letters?) You can use VS.net from the keyboard same as you can vi, but VS will expose 100x the features and the fraction of features you'll actually know how to use will also be far greater.

Of course none of that changes the fact that neither I nor you know how to do a good gui, but at least I don't try to esteem the limitations of my medium. The *nix crowd always does that, and it saddens me that *nix has hardly evolved in 30 years. 30 years!

On the other hand, though, I must say the command-line is very very useful when you're tying together various programs to do something new. Obviously that's the only way pdfrasterfarian was constructued, and I owe much gratitude. But if that's the rightful use of the command-line, then I think no one should have qualms with cramming in hundreds of options and parameters and a long manual that no human should ever have to use directly in the day-to-day.

And now, that's one opinion which I won't attempt to address at all. That's start another thread altogether, so I'd really prefer not to move off the topic we're discussing here

This was an issue of discussion in the backend thread, and it surprised me when you said "no, the backend shouldn't have so many command-line options."

alex_d, all I said was that "I don't want to add too many options unrelated to PDFRead (as the ones related to unpaper would be)". I don't think that's being unreasonable. To put it in context, would you replicate all the options in any GUI in a seperate GUI, then call the earlier GUI with the options filled in the second one? That's what I was objecting to, adding all the unpaper command-line options (and they are quite a lot) to satisfy maybe < 20% of the users. I have no problems in adding unpaper related functionality, but not in a way which would make it overly complex for everyone ie. use the principle "Make things as simple as they can be, and no simpler".

[nekokami]

Would it be possible/feasible to have an unpaper CLI option that takes unpaper arguments and passes them straight through?

Speaking as an 11-year employee of Sun Microsystems, I think Unix has come quite a long way in the last 30 years. I can now recommend Ubuntu, for example, to my non-geek friends, with a straight face. Never mind OSX, which is unix under the hood, and is being sold by Apple, the company that thinks you shouldn't need to know what's really going on with your computer. (I'm typing this on OSX.)

[ashkulz]

Would it be possible/feasible to have an unpaper CLI option that takes unpaper arguments and passes them straight through?

Now, that's something which is quite feasible. I'll have to study it a bit though because if you use unpaper to split a double page layout, then more pages will have to be created. The single page layout should be very easy to implement. nekokami, do you have a few samples of such scanned pages? I don't have a scanner, so I can't really test it out

I'll have to think of something to do in the Windows GUI. Is anyone volunteering to create one, even after hearing the opinions of alex_d and me?

[dstampe]

What kind of file vs. Reader space sizes can be expected? I tried converting a 3300-page PDF file (large print) and got a 100 MB LRF file! I thought file sizes were supposed to be comparable to the source PDF file? What kind of compression is used (if any)?

[ashkulz]

Originally Posted by dstampe

What kind of file vs. Reader space sizes can be expected? I tried converting a 3300-page PDF file (large print) and got a 100 MB LRF file! I thought file sizes were supposed to be comparable to the source PDF file? What kind of compression is used (if any)?

That's quite a lot of pages! Must have taken ages to convert, too.

File sizes will be comparable to the source PDF if it mostly consisted of graphics, otherwise it can be expected to be in the order of 2-3 times the PDF. However, I've tried this mostly with small PDFs (300-400 pages). Some possibilites on reducing file size:

• reduce the colors used to 2 (ie. monochrome). You can do this currently by customizing the profile
• Use the portrait mode (should result is lesser pages)
• optimize the PNG using optipng (extra processing step, not yet implemented)

Can you try #1 and #2 above?

[dstampe]

The file was already portrait. I tried a similar file with only 20 pages: 600K with 4 colors, 400K with 2 colors. I don't expect adding extra PNG processing is going to save more than 20% on file size.

Original PDF file size was 130K for the 20-page file, and 7500K for the 3300-page file. Both had embedded fonts. The 20-page file increased in size by a factor of 5, the 3300-page file by a factor of 13. Not sure why the uncrease is size was different in each case, as settings were the same.


Is there any way to disable dilation and resizing in order to keep fonts cleaner? Without using 4 grayscale levels, the resizing after dilation resukts in significantly messier characters. I'm not sure if the Reader does antialiasing of fonts internally utilizing the grayscale capability of the display.

[alex_d]

30-50 kb/page is the unavoidable result of converting to images. The difference in the "increase factor" simply has to do with how much non-text data the pdf contained. Output size stays consistent, input size can vary widely. Most people, however, find that 30-50 kb/page is less than what their original pdf took up. Nevertheless, a completely text pdf might use just a few kb per page.

These days, however, a 1GB sd card costs about $10 (with 4GB also available) so none of the above should really matter.

About the blurry text: Ashkulz chose not to implement the sharpening filter found in PDFrasterFarian. You might want to try the other program and see if you like the result. The possibilities for post-processing, however, are vast and few have so far been explored. If you have any suggestions, they would be welcome.

[dstampe]

Yes, memory cards are a possibility. The reason I don't use them for books is that it appears you can't organize the stuff on the card by collections. I tend to keep 100 or so books on the reader (just a preference). Another disadvantage is that the battery life is considerably shorter when reading from a card.

I wasn't really concerned about blurriness, in fact this is required for antialiasing. I wanted to eliminate dilation (and possibly resizing) because the low-vision fonts I use are already fat, and the extra dilation makes then merge too much.

Don't worry too much about these items, I am still searching for a way to handle text books with large low-vision fonts on the reader. So far I've tried:

- PDF with embedded fonts (works, as long as it's less than 200 pages or page turns get ridiculously long, and there is a hard 1000-page limit)

- PDF with conversion to graphics (slow conversion and large file sizes)

- BokkDesigner and embedded fonts (half works, but does not properly handle boldface and does not support left-justification and hyphenation). The result is poor readability with large fonts.

I am considering these as possible solutions:

- re-flashing the reader with better fonts and standardizing on RTF (main problem is getting good fonts that work with other books, and the lack of hyphenation).

- Trying to use Word to force hyphenation, then create hard page-breaks for left-justification. Hard to see how this can work as the font layout in Word and reader will probably differ.

[alex_d]

One way to organize on memory cards is to use the "author" field in pdfread or pdfrasterfarian as categories. You can then sort by "author" and get some semblance of organization. You can also carry multiple sd cards. Kind of fun and old school.

I'm not so sure about battery life being a lot shorter with a memory card, however. There might be an effect but nothing that would turn the reader's huge battery life into a short one.

Also, if you don't want any dilation, pdfrasterfarian has options to only do sharpening or to skip all postprocessing.

p.s. this probably isn't any of my business, but keep in mind that "going easy" on your eyes will, like not exercising any other part of your body, just make them go soft and deteriorate quicker. i mean if you can't read you can't read, but don't willfully overcompensate.

[sputnik]

It would be nice to have an option to select a range of pages to be converted, instead of the whole document. I got some very long pdf books (almost one thousand pages each) and conversion takes too long. I know, I could use Adobe products to obtain smaller documents (book chapters or sections), but a feature that you could use to specify a page range would be great.