k2pdfopt: optimizes PDFs for viewing on e-readers

[pdurrant]

Academic papers are often still in copyright. This one even had a copyright notice on each page. Removed. Please be more careful about uploading.

[willus]

Originally Posted by pdurrant

Academic papers are often still in copyright. This one even had a copyright notice on each page. Removed. Please be more careful about uploading.

My attachments replaced with small samples (less than one page of the source document).

[Shohreh]

I find sans serif fonts easier to read on my e-reader.

Unless I missed it, it looks like k2 doesn't support setting the fonts in the output.

Is there a utility that can replace them in the input file?

For instance, here are the fonts in a book I'm currently reading:

.SFNS-Regular_wdth_opsz110000_GRAD_wght (TrueType; embedded)
.SFNS-Regular_wdth_opsz110000_GRAD_wght (TrueType; Roman; embedded)
Alegreya-Regular (TrueType; Roman; embedded)

Thank you.

[Shohreh]

I have another question.

The original PDF is an image-only file — meaning: In SumatraPDF, I can't select text with the mouse.

After running that original file through k2, the output on my e-reader is fine.

However, I'm having a problem with inputs that I ran through Abbyy Fine Reader:
- Saved as "searchable" PDF: The output from k2 inserts carriage returns about half-way through each line
- Saved as "image" PDF: I only get empty pages.

I zipped a sample of each file (original, searchable, image). I used the following commands:

#mt to remove header
#good
k2pdfopt.exe -w 758 -h 1024 -dpi 213 -mode fw -ls- -mt 1.02 -fc- input.original.pdf
#wrong carriage returns
k2pdfopt.exe -w 758 -h 1024 -dpi 213 -mode fw -ls- -mt 1.02 -fc- input.OCR.searchable.pdf
#only empty pages
k2pdfopt.exe -w 758 -h 1024 -dpi 213 -mode fw -ls- -mt 1.02 -fc- input.OCR.image.pdf

Thank you.

[rkomar]

Originally Posted by Shohreh

I find sans serif fonts easier to read on my e-reader.

Unless I missed it, it looks like k2 doesn't support setting the fonts in the output.

Is there a utility that can replace them in the input file?

For instance, here are the fonts in a book I'm currently reading:

.SFNS-Regular_wdth_opsz110000_GRAD_wght (TrueType; embedded)
.SFNS-Regular_wdth_opsz110000_GRAD_wght (TrueType; Roman; embedded)
Alegreya-Regular (TrueType; Roman; embedded)

Thank you.

It is generally a bad idea to replace the fonts in a PDF. When the PDF is constructed, the characters are precisely positioned (i.e. kerned) according to their shapes in the original font. If you change the font, then you get weird spacings between the characters in words (some have bigger gaps, others overlap). It just looks bad.

EPUBs are rendered completely differently from PDFs. There is no kerning information in EPUBs, so swapping fonts doesn't lead to weird character spacing.

[Shohreh]

Too bad. Thanks.

I'm struggling at getting k2 to convert a searchable PDF to something that displays on my reader as well as it does on the computer. For some reason, the output has carriage returns in the middle, along with garbage characters like #, and the font size changes occasionally. Unreadable.

I'm suprised since the doco says that by default, k2 converts PDFs to bitmaps so the output should be clean without having anything fancier to do than this:

k2pdfopt.exe -w 758 -h 1024 input.pdf

---
Edit: Using the GUI, I found that the solution is to keep the "Re-flow text" unchecked (default: -fc- -wrap-), which looks better than checking that option (-fc- -wrap+)… and way better than the options I tried through the CLI after reading the list of options availables.

Edit: False hope. I tried again, and for some reason, I get crap, with or without the wrap option (-fc- -wrap-, -fc- -wrap+). I don't get it.

--
Edit: As a work-around, I convert the PDF into PNG, merge them back into a PDF, and run it through k2

#Use best resolution
mutool.exe convert -O resolution=600 -o %03d.png input.pdf

FAILS mutool.exe merge -o output.pdf -O compress *.png
img2pdf --title "My book" --author "My author" -o output.pdf *.png

k2pdfopt.exe -odpi 213 -m 0.2 -fc- -wrap- -h 1024 -w 758 output.pdf

Still worth it because, like the help page says, bitmaps are displayed faster than a native PDF, besides the optimized output for 6" readers.

[galloshumour]

Does anyone have tips for processing magazine PDFs with k2pdfopt? I've been trying to figure out how to best read a 3-column magazine that has text interspersed with lots of images and small cartoons, but the text gets broken up oddly around the images/cartoons

[willus]

Originally Posted by Shohreh

...
However, I'm having a problem with inputs that I ran through Abbyy Fine Reader:
- Saved as "searchable" PDF: The output from k2 inserts carriage returns about half-way through each line
- Saved as "image" PDF: I only get empty pages.

I cannot reproduce what you describe. Are you running k2pdfopt v2.55 on Windows. I've attached a couple of pages of output from your 2nd and 3rd commands (I added -p 1-2 to them). They all look the same. Is it only on your e-reader that the PDFs display funny? I'm just looking at these using Sumatra PDF on my desktop PC. There's really no way k2pdfopt can be inserting a carriage return in "fitwidth" mode. It's just adding commands to the source PDF to display different cropped regions of the source content on each page. It's not doing any text re-flow. I'm guessing this is an artifact of whatever reader you are using.

BTW, you used # for comments in your command list, so clearly you're not running them in a .bat file. What shell are you using?

[willus]

Originally Posted by rkomar

It is generally a bad idea to replace the fonts in a PDF. When the PDF is constructed, the characters are precisely positioned (i.e. kerned) according to their shapes in the original font. If you change the font, then you get weird spacings between the characters in words (some have bigger gaps, others overlap). It just looks bad.

EPUBs are rendered completely differently from PDFs. There is no kerning information in EPUBs, so swapping fonts doesn't lead to weird character spacing.

Agree 100%. I was going to write the same thing but you beat me to it. Thank you, rkomar.

[willus]

Originally Posted by galloshumour

Does anyone have tips for processing magazine PDFs with k2pdfopt? I've been trying to figure out how to best read a 3-column magazine that has text interspersed with lots of images and small cartoons, but the text gets broken up oddly around the images/cartoons

It all kind of depends on the specific 3-column layout, but I guess I'd try to set k2pdfopt for > 2 columns (-col 4) and then you may have to tweak some of the column finding parameters. Search "column" in the command-line params or maybe check this somewhat dated help page on column detection. If you could post a couple of pages or privately mail me a link to your doc I might be able to suggest something more specific.

[Shohreh]

Thansk for the help.

Originally Posted by willus

I cannot reproduce what you describe. Are you running k2pdfopt v2.55 on Windows.

Yes.

Originally Posted by willus

Is it only on your e-reader that the PDFs display funny?

Indeed, I now suspect it's just my cheap and older 6" reader whose PDF reader isn't up to par. I don't have another e-reader to check. The k2 output looks fine on SumatraPDF.

Originally Posted by willus

BTW, you used # for comments in your command list, so clearly you're not running them in a .bat file. What shell are you using?

Plain Windows. The # were just to show they're comments :-)

Since then, I noticed Abbyy Finereader can convert a PDF to EPUB… with amazing result.
For "bitmap PDFs" that also contain an OCR layer , I'll just have to find a good English/French spellchecker to fix the few typos in the HTML files in the EPUB output.

[Used1]

Originally Posted by willus

This is why it really helps to see the source file. It has text along the bottom left edges, and that's causing problems. You have to crop out that area of the page. Fortunately you have a nice clean document.

Conversion #1
See k2pdfopt_window.png and cropregion_window.png attachments.
Here I tried to stay as close to your options as possible
1. I selected a crop region to ignore the edges of the page.
2. I selected "Erase horizontal lines" to ignore the divider lines on some of
the pages.
3. I de-selected auto-crop. That's really meant to be for poorly scanned
books that have photocopying artifacts at the edges of the pages. It doesn't
work very well.
4. I set max columns to 1 just so k2pdfopt would not look for multiple columns
5. I put -fc- in the additional options box to suppress a warning message.

It's also important that "Autostraighten" is selected. I set the font size to
18 points. I attached sample output to output_sample_18ptfont.pdf. The conversion was also pretty slow--200 s on my PC. See below for how I remedied that.

Conversion #2
(k2pdfopt_window_faster.png and
output_sample_faster.pdf)
Here I made some changes to make the conversion faster and the output file smaller. Rather than setting the output font size, which is slow for a number of reasons (part of it that k2pdfopt has to analyze the file to determine the font size), I just tell k2pdfopt that your kindle window size is much smaller than it actually is. It seems strange, but this has the same effect, and k2pdfopt processes everything at a lower dpi, so it goes much faster and makes a smaller output file. Also, I set the input dpi to 300 dpi since that is the native resolution of the source file. Normally it defaults to twice the output dpi. Finally, I added "-rt 0" so k2pdfopt does not waste time verifying the orientation of the source file. This brought the conversion time down from 200 s to 45 s on my PC. The output file size is ~11 MB.

Conversion #3
(k2pdfopt_window_fastest_and_smallest.png and output_sample_fastest_and_smallest.pdf)
I added -bpc 1 to the additional options since the source document is just a monochrome (1-bit) document. This gets the output down to ~7 MB from an original size of ~20 MB in conversion #1.

Apologies for the delay. Life has been busy. Thank you for taking the time to walk me through this. I'll continue playing with it and see where I have further questions. The explanation of the features was very helpful.

[hardreader]

Hi folks, I'm new around these parts and could use some help.

I have an issue that seems basic compared to some of the ones everyone is experiencing, but I've been banging my head against the wall for a long time poring through the command-line options documentation and everything. Maybe someone can help.

My pdf has page numbers and chapter names in the headers that I would like to keep at the top of the page, but no matter what options I choose, I get stuff like this:



This was output just in default mode with my Paperwhite SE dimensions as -h and -w. I made sure to use -pb+. I can't remember whether I had -as enabled, but I don't think that should affect this. My understanding is that -pb+ should be inserting a page break between (for example) the end of page 16 and the header on page 17, which you can see right in the middle of the page.

I've also tried using the calibre plugin interface with no success. Please help!

[hardreader]

Originally Posted by hardreader

Hi folks, I'm new around these parts and could use some help.

I have an issue that seems basic compared to some of the ones everyone is experiencing, but I've been banging my head against the wall for a long time poring through the command-line options documentation and everything. Maybe someone can help.

My pdf has page numbers and chapter names in the headers that I would like to keep at the top of the page, but no matter what options I choose, I get stuff like this:



This was output just in default mode with my Paperwhite SE dimensions as -h and -w. I made sure to use -pb+. I can't remember whether I had -as enabled, but I don't think that should affect this. My understanding is that -pb+ should be inserting a page break between (for example) the end of page 16 and the header on page 17, which you can see right in the middle of the page.

I've also tried using the calibre plugin interface with no success. Please help!

Sorry, link issue. See image here:

[willus]

If I'm interpreting correctly, you want to put each source page header at the top of each destination page--basically you want to preserve page headers/footers. I don't recall anybody ever asking for that before, surprisingly (because it's a very reasonable ask). Presently there is no way to do that with k2pdfopt. I'll add it to my feature request list, but I haven't had time to do a major update in quite a while.