(sigh), Kindle/KPF format Blues...

[Hitch]

So, sportsfans:

Another happy week in the world of eBookmaking. I have a RAVING client, who is irate about the service we've provided him. That's of no interest to anyone here but me, but, in terms of me kvetching, for a change, about Amazon, and why they cannot make up their damn minds and stop screwing those of us in the biz:

As some of you may remember, when Textbook Creator first came out, you built a .kpf file and you could build a "print replica" (herein, "PR") eBook that would work as a fixed-layout "mobi" (I'll get back to that nomenclature in a moment) that could be bought on eInks, as well as Fires.

Alrighty, well, fast-forward some amount of time. First, we determine that internal (not external) links no longer work in PR. At first, I thought it was a glitch, something we'd done on an eBook and it took months to get Amazon to focus on it, as they kept assuming (as we were handed along from one person to the next) that we were stupid, and that we meant external links on an iOS device, which, as we all know, haven't worked in what, 7 years now?

But finally, they 'fessed up--the internal links (TOCs, Indices, any internal links) just don't work, period, other than K4PC/K4Mac. Oh, joy. That was 6 months ago. They said that they were working on it, so...presumably, it would be fixed by now, right? (NO is the answer, in case you foolishly thought it was yes, as I did.)

Then, I have a client that's read this Format Comparison Chart: https://kdp.amazon.com/en_US/help/topic/G202101750 (scroll down if that opens at the top of the page) and decides that PR is just the thing he wants.

We discuss PR and I say, in my email to him, that PR only works on the "more advanced Kindle devices." I did not say "NOTHING BUT FIRES," because that's not quite accurate.

So, we make the file. First, he's unhappy because it doesn't "snap" to the page, in spread view, as does iBooks with FXLs made in spread mode, so he wants us to remake it in one-page up view. Then something else happened and we had to remake the index, a 2nd time...fine. I don't charge for any of this rework.

He takes the file and publishes it.

What happens? First thing, a friend of his buys it and said friend has an eInk, a PPW, and he can't read it, because it's not allowed for his device. Add to that that the client buys it for his own device (K4iOS/iPad) and of course, now, the internal links do not work, EITHER.

So, he's irate. He writes to KDP and one of their CSR's sends him a canned email, telling him which devices it can work on, and HELPFULLY (NOT!) adds as a last paragraph:

If you want readers from paper-withe devices to see your book , you can publish your book with a MOBI file. Kindle Direct Publishing (KDP) lets you convert your content from several formats. However, for best results we recommend you upload your content in Microsoft Word, MOBI, or Zipped HTML format.

(BOLD emphasis added)

So, now, he's decided that I've ripped him off. That .kpf is some rogue format, akin to what happens if you buy Chinese car replacement parts, instead of Ford-issued parts, or whatever. I've explained repeatedly that it's an authorized Amazon format, yadda. But of course, he won't hear it. His position is, that because I said "MOBI" repeatedly (which I did--when you say "kpf" or "kcb" or whatever, to people, they typically get all worked up that you're giving them the "wrong" file!), but of course, you guys know me, it wasn't my intent to mislead him. I mean, WHY would I do that?

So anyway--the latest and "greatest" (so to speak) on PR is, internal links do not work on anything, ANYTHING, but K4PC and K4Mac and they are no longer buyable on eInks. Apparently, I can only assume that Amazon has decided that the eInks, even the latest voyages, PPWs, etc., just don't have the brainpower (processing power) and all that to handle PR or other FXL eBooks.

So, if you deal with clients, and you haven't been minutely detailed about the limitations of PR, you've been warned. Make sure you develop religion about saying "this doesn't work and that doesn't happen."

Hitch

[JSWolf]

Hitch, that was a rather interesting read.

What I want to know is why is that that some who want an eBook made don't listen to those who know what they are talking about? Fixed layout was a bad idea for ePub3 and it's a bad idea for Amazon.

You get people thinking their eBook would be best in fixed-layout when really, that's the worst idea ever. The problem is that the standards committee decided to let Apple dictate to them what would be good for ePub3. And what's good for Apple is usually bad for everyone else. Of course, Amazon has to have their own version of ePub3 aka KF8. And who at Amazon decided to allow the worst features of ePub3?

And because these features are part of the standard and even though they are so not supported, your clients see these features and go "Oooohhhh! Shiny feature. I want." (said in the voice of Homer Simpson) They don't listen when you say it's not a good idea and won't work well and your eBook won't sell. They don't listen until after you've put in the work to make it as the client wanted and then sees that you were correct.

So what really needs to happen is to take ePub3 and throw it away. It's garbage. This would mean that every ePub3 eBook made would have to be remade as ePub2. So be it to get rid of ePub3.

ePub4 needs to be created with features that that are not poorly implemented and that can be implemented on eInk Readers. And they need to leave Apple out of suggesting features for ePub4. ePub4 would be the chance to do it right instead of redoing ePub3 and making it even worse.

[Hitch]

Originally Posted by JSWolf

Hitch, that was a rather interesting read.

Yeah, I wish it were happening to someone else.

What I want to know is why is that that some who want an eBook made don't listen to those who know what they are talking about? Fixed layout was a bad idea for ePub3 and it's a bad idea for Amazon.

Wolfie, to be fair, this book can ONLY be made in FXL. There's no reflowable option for it. It's a textbook about drawing in pen and ink. There's no really good way to make it reflowable. You know me, we'd have done that were it remotely possible. The choice of FXL--that's not his mistake. It's fxl or naught.

You get people thinking their eBook would be best in fixed-layout when really, that's the worst idea ever. The problem is that the standards committee decided to let Apple dictate to them what would be good for ePub3. And what's good for Apple is usually bad for everyone else. Of course, Amazon has to have their own version of ePub3 aka KF8. And who at Amazon decided to allow the worst features of ePub3?

Well, Apple's influence at the Committee--yes, that was bad.

And because these features are part of the standard and even though they are so not supported, your clients see these features and go "Oooohhhh! Shiny feature. I want." (said in the voice of Homer Simpson) They don't listen when you say it's not a good idea and won't work well and your eBook won't sell. They don't listen until after you've put in the work to make it as the client wanted and then sees that you were correct.

I didn't tell him it was a bad idea, Wolfie. THAT part is not his fault, sugar.

So what really needs to happen is to take ePub3 and throw it away. It's garbage. This would mean that every ePub3 eBook made would have to be remade as ePub2. So be it to get rid of ePub3.

Wolfie, I think that your soapbox is starting to sag. ;-) There's nothing "wrong" with ePUB3. Yes, it has a litany of features that are...questionable, but there's no law that says that we, as bookmakers, have to use them. Part of our job is to steer clients away from bad choices. In my scenario, I didn't realize that the client had an idea about the job that wasn't going to happen. And I didn't realize (and should have asked, in hindsight) if the Zon had fixed the link issue. (sigh).

ePub4 needs to be created with features that that are not poorly implemented and that can be implemented on eInk Readers. And they need to leave Apple out of suggesting features for ePub4. ePub4 would be the chance to do it right instead of redoing ePub3 and making it even worse.

Weeeeeealllll, kiddo, while ePUB4 would be nice, I seriously, seriously doubt that they're going to take features AWAY. Right?

Hitch

[slowsmile]

@Hitch...I hope you don't mind me asking you some questions about the new Kindle format that are a little 'off-piste' but they do concern oddities or things that are somewhat unclear wrt KPF format.

I think it's pretty clear that Amazon Kindle have made some massive changes to their conversion platform at their end. And, as you've pointed out from their laissez-faire attitude towards the irksome problems with inoperable links in print replica or fixed format, they are not there yet. I'd like to add to this with several questions.

For reflowable ebook uploads only, is it best to upload an epub(with ET enabled) or a KPF file to KDP? The reason I'm asking is that if you upload an Epub, I've assumed that the epub will still be converted to mobi, KF8 as well as KFX format at the Amazon Kindle end whereas if you upload a KPF file then it will only display on KFX enabled devices only(and not on older mobi or older KF8 devices that don't have ET capability). I could be way out of the ballpark here but would still love to know the answer to the above question. And do you still upload reflowable ebooks to KDP as epubs?

Second question. Part of the ET specification mentions the new Progressive Download capability for KFX files, which allows the purchaser of any KFX or ET-enabled ebook from Amazon to actually start reading their ebook while its downloading. So, since file size and download time are now completely irrelevant for KFX ebooks, why the heck is Kindle still charging a download penalty of $0.15 per Mb for file sizes over 5 Mb for those ET-enabled ebooks??

Last question. How do you estimate the file size of a KPF file?? This is particularly important if you've created an image-dense ebook or fixed format ebook using Kindle Create and you wish to minimize or avoid Kindle's download penalty. And I realiize that this is, perhaps, an unfair question for you because I also know that you don't use Kindle Create. But perhaps someone else could help on this forum. Since KFX appears to be yet another proprietary epub format from Amazon, I currently estimate KPF file size by converting to epub using JP Howell's KFX Input plugin and then take the file size of the epub. Yes, I know that this method of estimating will only give me a very rough ballpark estimation of file size. But there seems to be no other way of doing it. I'ld really prefer a more accurate way of assessing the file size of KPF files before upload.

[Hitch]

Originally Posted by slowsmile

@Hitch...I hope you don't mind me asking you some questions about the new Kindle format that are a little 'off-piste' but they do concern oddities or things that are somewhat unclear wrt the KPF format.

You can ask...I may not have very good answers.

I think it's clear that that Amazon Kindle have made some massive changes to their conversion platform at their end. And, as you've pointed out from their laissez-fair attitude towards the irksome problems with links in print replica or fixed format, they are not there yet. I'd like to add to this with two questions.

For reflowable ebook uploads only, is it best to upload an epub(with ET enabled) or a KPF file to KDP? The reason I'm asking is that if you upload an Epub, I've assumed that the epub will still be converted to mobi, KF8 as well as KFX format at the Amazon Kindle end whereas if you upload a KPF file then it will only display on KFX enabled devices(and not on older mobi or KF8 devices that don't have ET capability). I could be way out of the ballpark here but would love to know the answer to this question. Do you still upload your ebooks to KDP as epubs?

ACTUALLY, we still give our customers made-to-order MOBI files. I mean, sure...if a customer is having a cow, because they read some blog somewhere, we'll give them (solely) an ePUB, but generally speaking, we don't trust the auto converter. (Plus, don't forget, we do a TON of font work!) We build and test the MOBIs ourselves. We don't believe in the "one file to rule them all" approach, e.g., give the client an ePUB and turn 'em loose.

When you do that, you can't use media queries for the Kindle versions. We prefer to optimize the files for the Kindles, both new and old. (I just updated my K2, for crying out loud and it still works, chugging along...)

I mean...if we're asked for a single ePUB, and no mobi, we can and will do that but our preference is to optimize the files for the environment to which they are destined.

Like you, I'm concerned about the realities, behind the scenes, with KPF uploads. Not that my job is to think about folks who might buy a book and want to download it and er, tweak it or whatever, but...just the same...

I like sticking with stuff that I can dissect, if need be. Take apart, find out what's wrong, see the code...all that.

Second question. Part of the ET specification mentions Progressive Download capability for KFX files, which allows the purchaser of any KFX or ET-enabled ebook from Amazon to actually start reading their ebook while its downloading. So, since download time is now irrelevant for KFX ebooks, why the heck is Kindle still charging a download penalty of $0.15 per Mb for file sizes over 5 Mb for those ET-enabled ebooks??

Well...the progressive part doesn't eliminate the bandwidth usage, though, does it? Or do I just not understand something here? I freely admit, I've never had much of a wait time for a Kindle eBook to download and I live in the Far Boonies. So...I never thought about it.

I truly don't know how PD (Progressive Download) affects bandwidth usage. Maybe one of the Nerdier Dudes here can speak to that?

Hitch

[slowsmile]

ACTUALLY, we still give our customers made-to-order MOBI files. I mean, sure...if a customer is having a cow, because they read some blog somewhere, we'll give them (solely) an ePUB, but generally speaking, we don't trust the auto converter. (Plus, don't forget, we do a TON of font work!) We build and test the MOBIs ourselves. We don't believe in the "one file to rule them all" approach, e.g., give the client an ePUB and turn 'em loose.

When you do that, you can't use media queries for the Kindle versions. We prefer to optimize the files for the Kindles, both new and old. (I just updated my K2, for crying out loud and it still works, chugging along...)

I mean...if we're asked for a single ePUB, and no mobi, we can and will do that but our preference is to optimize the files for the environment to which they are destined.

Like you, I'm concerned about the realities, behind the scenes, with KPF uploads. Not that my job is to think about folks who might buy a book and want to download it and er, tweak it or whatever, but...just the same...

I like sticking with stuff that I can dissect, if need be. Take apart, find out what's wrong, see the code...all that.

@Hitch...Alot of people on the KDP forum seem to think that I'm in love with Kindle Create. I do like the app -- but with many frustrated caveats. I'll give you one simple example. I'm writing a how-to ebook on KC and to me it's absolutely essential that there is a multi-level NCX TOC in my Kindle ebook to help the reader easily navigate all the heading levels in the ebook. I mean it's a how-to book for goodness sake where readers will always need to quickly jump around to reference particular bits of information. But how can you do that easily when Kindle Create will only ever give you a single level NCX TOC?? I mean you can even create multi-level NCX TOCs easily in Word or epub that will work just fine in Kindle ebooks. So why the hell can't you do that in Kindle Create? I've already suggested this twice to dev support as a change to KC's NCX TOC and I'm giving them till Jan 2020 to make and release that change so I can actually use Kindle Create to produce that ebook. If it doesn't happen then I'll just use Jutoh with Sigil to produce the epub versions(with a multi-level NCX TOC) and Kindle Create can go swing.

And a big thank you for the above information. It's funny because in the Kindle documentation they brag that the command line version of the new Kindle Previewer is suitable for batch processing KPF files for professional formatters. And here you are as a professional formatter saying you wouldn't touch Kindle Create or KPF with a barge pole. So I doubt if any professional formatter worth his/her salt is using that command line utility to produce any KPF files. I also understand why as well -- because KC doesn't give you enough control over the formatting + distribution problems.

You've also given me a great idea. I think I'll add a new chapter to my how-to book on KC called "Kindle Create: Critique, Caveats and Wish List" where I could vent all my pent up frustrations about KC. Seriously. It might wake them up.

[jhowell]

You addressed your questions to Hitch, but I think I can provide some of the answers for you.

Originally Posted by slowsmile

For reflowable ebook uploads only, is it best to upload an epub(with ET enabled) or a KPF file to KDP? The reason I'm asking is that if you upload an Epub, I've assumed that the epub will still be converted to mobi, KF8 as well as KFX format at the Amazon Kindle end whereas if you upload a KPF file then it will only display on KFX enabled devices only(and not on older mobi or older KF8 devices that don't have ET capability).

Reflowable books published using KPF files created using KC will be delivered to customers in KFX format with Enhanced Typesetting when possible. When KFX is not possible, due to an older device or using download & transfer via USB, MOBI7 format will be used instead. Amazon derives that from the KPF file.

No KF8 format is produced for these books as far as I can tell in my testing. I assume that the same will also be true of books published using KPF files from Kindle Previewer.

Originally Posted by slowsmile

Second question. Part of the ET specification mentions the new Progressive Download capability for KFX files, which allows the purchaser of any KFX or ET-enabled ebook from Amazon to actually start reading their ebook while its downloading.

The entire book will download as quickly as possible. The difference is that the customer can usually begin reading before it completes.

I do think that Amazon's download charge is excessive given the major declines in the prices of bandwidth and storage over time.

Originally Posted by slowsmile

Last question. How do you estimate the file size of a KPF file??

It don't know of any good way to do that. The size of the KPF file could be used as a rough upper limit.

[Tex2002ans]

Originally Posted by jhowell

[...] When KFX is not possible, due to an older device or using download & transfer via USB, MOBI7 format will be used instead. Amazon derives that from the KPF file.

No KF8 format is produced for these books as far as I can tell in my testing.

Holy crap, that sounds horrible!

Originally Posted by Hitch

Weeeeeealllll, kiddo, while ePUB4 would be nice, I seriously, seriously doubt that they're going to take features AWAY. Right?

lol, 3.2 acted like 3.1 never even existed.

If JSWolf had his way, EPUB4 would act like EPUB3 never existed (but that won't happen).

There's a lot of good things in EPUB3 (semantics, MathML), but a lot of bad things too.

Originally Posted by jhowell

The entire book will download as quickly as possible. The difference is that the customer can usually begin reading before it completes.

... and honestly, how often would this even happen? Aren't most ebooks only a few MBs?

Even in the case of a larger book (picture book, etc.), it would only help with extremely slow internet.

I guess people really want to read NOW NOW NOW! Instead of a few seconds from now.

[slowsmile]

"No KF8 format is produced for these books as far as I can tell in my testing. I assume that the same will also be true of books published using KPF files from Kindle Previewer."

@Jhowell...Does what you say also mean that if a person with an older KF8 device(without ET capability) buys a KFX ebook then or she is going to get a crappy mobi7 file instead?? I always thought that if ET was not enabled and you formatted your epub as KF8 then you would just get KF8 without ET enabled.

And what about if you upload an epub direct to KDP that's formatted for both KF8 and KF7 with ET enabled in testing. If I upload that epub to Amazon Kindle will it take a different conversion route and be deliverable to KF7, KF8 and KFX devices as needed or not??

And, from what Hitch has already said, the only way you can successfully target your epub for all KF7 and KF8 devices without problems -- which would also include KF8 devices with KFX capability -- would seem to be by just formatting your epub as a dual mobi using kindlegen and uploading that to KDP. Have I got that part right?

[jhowell]

Originally Posted by slowsmile

@Jhowell...Does what you say also mean that if a person with an older KF8 device(without ET capability) buys a KFX ebook then or she is going to get a crappy mobi7 file instead??

It depends on the format the publisher uploads to Amazon. Only KFX or MOBI7 will delivered to customers if reflowable KPF is used.

Originally Posted by slowsmile

And what about if you upload an epub direct to KDP that's formatted for both KF8 and KF7 with ET enabled in testing. If I upload that epub to Amazon Kindle will it take a different conversion route and be deliverable to KF7, KF8 and KFX devices as needed or not??

Uploading an EPUB, DOCX, or Master MOBI will result in MOBI7, KF8 and usually KFX.

Originally Posted by slowsmile

And, from what Hitch has already said, the only way you can successfully target your epub for all KF7 and KF8 devices without problems -- which would also include KF8 devices with KFX capability -- would seem to be by just formatting your epub as a dual mobi using kindlegen and uploading that to KDP. Have I got that part right?

You can target different Kindle formats using media queries in EPUB to adjust the rendering of books.

You can just upload the EPUB to Amazon and they will produce the Kindle formats from it. You don't need to use kindlegen other than for testing.

[JSWolf]

Originally Posted by Tex2002ans

If JSWolf had his way, EPUB4 would act like EPUB3 never existed (but that won't happen).

There's a lot of good things in EPUB3 (semantics, MathML), but a lot of bad things too.

The thing to do woould be to forget ePub3 and work on doing ePub4 correctly and anything good from ePub3 would be done right (if it wasn't done correctly). The thing is all that crap that Apple added in and the way some things are done really need to go. So yes it would be easier to screp ePub3 and move to ePub4.

[Hitch]

Originally Posted by slowsmile

And, from what Hitch has already said, the only way you can successfully target your epub for all KF7 and KF8 devices without problems -- which would also include KF8 devices with KFX capability -- would seem to be by just formatting your epub as a dual mobi using kindlegen and uploading that to KDP. Have I got that part right?

Our belief at BN.biz is that this is right--but you should be able to upload the ePUB, or ePUB w/media queries, or as Jhowell says, Word file or Zipped HTML package, I believe, should result in the KF7, KF8 and the KFX, afaik. That's what our testing seems to indicate. I'll defer to jhowell's superior knowledge--he seems to do a LOTTA testing and honestly, we do what we can but obviously our primary focus is on delivering eBooks to our customers.

Hitch

[slowsmile]

@Hitch... Thanks to you and @JPHowell for confirming that for me -- I really wasn't sure if I had it right or not. I now know how to format my ebooks for all Kindle Formats.

[slowsmile]

From what both JPHowell and Hitch have said, if you upload a KPF file it will either be converted to KFX or MOBI7 -- which will only work properly on the more recent ET-enabled KF8 devices and on the older MOBI7 devices. And older KF8 devices that do not have ET capability will just receive the MOBI7 version.

So here's another interesting question. So what if you upload an image-dense KPF as a reflowable ebook(KFX) and a person buys your ebook who has an older KF8 device without ET capability. He will get the MOBI7 version, right? Which means that all the images from your ebook on that person's KF8 device will look utterly crappy(images probably way too large!). So he complains on your product page or complains directly to Amazon, who then send you a KQN about those crappy images in your ebook. I mean what can you do about that, particularly when it isn't really your fault -- it's really Amazon's fault for causing that incompatibility in the first place? That's why I think many people who produce image-dense ebooks using KPF format will be getting KQNs. And I doubt if many of those indie publishers with that problem will know what to do to fix it because this problem and its solution aren't documented or mentioned anywhere in the current Kindle docs as far as I know.

[jhowell]

Originally Posted by slowsmile

So here's another interesting question. So what if you upload an image-dense KPF as a reflowable ebook(KFX) and a person buys your ebook who has an older KF8 device without ET capability. He will get the MOBI7 version, right? Which means that all the images from your ebook on that person's KF8 device will look utterly crappy(images probably way too large!).

A lot depends on how good of a job Amazon does in producing the MOBI7 format of the book. If you are that concerned about the results, you could publish a sample book with various sized images created using KC and see how it looks on an older Kindle.