Amazon removed books from my Kindle

[JSWolf]

Do you loose any features when an eBook is set to PDOC vs EBOK?

[Mudchild]

Hi all, new here, and am wading through all this new info to try and make sense of what happened - my Kindle suddenly lost all the sideloaded books along with their precious highlights, after the on/off switch got trapped under the finger strap, causing multiple restarts...

However connecting to the computer shows that all the sideloaded books are still on the device, not wiped or removed - the Kindle just stopped seeing them somehow.

Also MyClippings is still up to date with the most recent highlights I made before the incident.

What I'm wondering is, is there any way to get the Kindle to see the books again, along with their highlights? I'm imagining that just dragging all the books back on from Calibre would probably mean I would definitely not see my highlights again...

Any pointers much appreciated!

[jlinear]

Originally Posted by Sirtel

AFAIK the book may well be on your account, but it will be removed if it's not supposed to be on that specific device. For example, deDRMed and sideloaded Amazon purchases would be removed, unless Download & Transfer was used for that specific Kindle.

This was not true in my case. Books purchased from Amazon, targeted to my Paperwhite 11th gen, deDRMed and sideloaded by Calibre to BOTH my Paperwhite and Scribe, were deleted from both upon each connecting to WiFi after a period in Airplane mode. I did not download each book twice to target each device. Target device was immaterial in my case.

[Sirtel]

Originally Posted by jlinear

This was not true in my case. Books purchased from Amazon, targeted to my Paperwhite 11th gen, deDRMed and sideloaded by Calibre to BOTH my Paperwhite and Scribe, were deleted from both upon each connecting to WiFi after a period in Airplane mode. I did not download each book twice to target each device. Target device was immaterial in my case.

Strange. This raises the question, what are the criteria Amazon bases their deleting on? If the book exists on the account and is even supposed to exist on the Kindle it was deleted from, WHY was it deleted?

[j.p.s]

Originally Posted by Mudchild

Hi all, new here, and am wading through all this new info to try and make sense of what happened - my Kindle suddenly lost all the sideloaded books along with their precious highlights, after the on/off switch got trapped under the finger strap, causing multiple restarts...

However connecting to the computer shows that all the sideloaded books are still on the device, not wiped or removed - the Kindle just stopped seeing them somehow.

The entries in cc.db for the books got changed.

One column is named something like is_visible.

Also MyClippings is still up to date with the most recent highlights I made before the incident.

MyClippings.txt is just a log file. The kindle only ever writes to it, never reads. The loacations of the highlights for a book are stored in a file in the .sdr directory for that book.

What I'm wondering is, is there any way to get the Kindle to see the books again, along with their highlights? I'm imagining that just dragging all the books back on from Calibre would probably mean I would definitely not see my highlights again...

Copy the entire documents directory on the kindle to a computer, then delete or move all the books and .sdr directories in the documents directory on the kindle. Then copy the books and .sdr directories from the computer back to the documents directory on the kindle over USB. Then safely eject the kindle. The books should show up as "new" and when you open them the highlights should be there.

If you use any other method to tranfer the books, the metadata might change in a way that the contents of the old .sdr don't correspond to the book files then they might not be recognized as matching.

Any pointers much appreciated!

If the kindle is jailbroken, it might be possible to manipulate cc.db to make the books show up again, which would be easier for some and next to impossible for most.

[haertig]

Originally Posted by Sarmat89

There is no way to create an AZW file which has the type 'EBOK' other than reverse engineering the resulting file to misrepresent it as an Amazon product.

I could take a can of spray paint and scrawl "Ferrari" on the side of my car too. That doesn't mean I'm "misrepresenting" anything, nor does it give Ferrari the authority to come take my car. And just who would I be misrepresenting something to by putting EBOK in the metadata of my computer file on my device? It's my book, on my Kindle - am I misrepresenting something to myself? Does Amazon have authority to correct any confusion that I may be imparting on my own self? By deleting something that I own none-the-less?

That is neither supported nor endorsed by Amazon.

Who cares if Amazon supports or endorses it? That's completely irrelevant.

[j.p.s]

Originally Posted by Sirtel

Strange. This raises the question, what are the criteria Amazon bases their deleting on? If the book exists on the account and is even supposed to exist on the Kindle it was deleted from, WHY was it deleted?

Kindles keep a database of all books "from" amazon with information for whether each book is "downloaded" or "in the cloud/library". Amazon servers have "content and devices" database(s) with similar information.

My guess is that synchronization of these databases is part of the book sync process. If you delete a book from a kindle via the "My Content and Devices" web page, the row for that book on the server database gets modified. If a kindle has the wrong time the order of operations gets muddled and the kindle mistakenly thinks a book has been deleted via "My C&D".

When all this got designed, written, and debugged, they probably never considered that EBOK would ever get set by anyone but amazon and also assumed that time would always be kept "close enough" by wireless or didn't even consider that bad time might cause a problem.

The above is not meant to be correct in detail or even in general outline, but to convey how things can go wrong with software or any process.

[slm]

About spray-painting "Ferrari" on your car. As this is a registered trademark, the owner of the trademark probably can require you to remove it and to pay for the use you have made of it.

[Sirtel]

Originally Posted by j.p.s

Kindles keep a database of all books "from" amazon with information for whether each book is "downloaded" or "in the cloud/library". Amazon servers have "content and devices" database(s) with similar information.

My guess is that synchronization of these databases is part of the book sync process. If you delete a book from a kindle via the "My Content and Devices" web page, the row for that book on the server database gets modified. If a kindle has the wrong time the order of operations gets muddled and the kindle mistakenly thinks a book has been deleted via "My C&D".

When all this got designed, written, and debugged, they probably never considered that EBOK would ever get set by anyone but amazon and also assumed that time would always be kept "close enough" by wireless or didn't even consider that bad time might cause a problem.

The above is not meant to be correct in detail or even in general outline, but to convey how things can go wrong with software or any process.

I could understand this if the time was wrong by, well, weeks or months. But others have experimented and found that deleting happens even when the time is wrong by a few minutes. Wouldn't that be "close enough"? I don't know, all this seems just weirder and weirder, and it's been going on for more than 2 years by now, as indicated by the posts in this thread. I'm not sure we'll ever see Amazon fixing it.

Is it even certain that not using the EBOK tag prevents the book from being deleted?

[j.p.s]

Originally Posted by haertig

I could take a can of spray paint and scrawl "Ferrari" on the side of my car too. That doesn't mean I'm "misrepresenting" anything, nor does it give Ferrari the authority to come take my car. And just who would I be misrepresenting something to by putting EBOK in the metadata of my computer file on my device? It's my book, on my Kindle - am I misrepresenting something to myself? Does Amazon have authority to correct any confusion that I may be imparting on my own self? By deleting something that I own none-the-less?

Who cares if Amazon supports or endorses it? That's completely irrelevant.

It's completely relevant to what kind of baked in assumptions are present in the firmware and server software. Whether any given assumption is stupid, justified, or in somewhere in between depends on point of view and other constraints and considerations. But you already know all that. Or do you think that perfection is not only possible but also required?

(My own opinion is that amazon's priority for quality is pretty low and its arrogance is pretty high.)

[haertig]

Originally Posted by slm

About spray-painting "Ferrari" on your car. As this is a registered trademark, the owner of the trademark probably can require you to remove it and to pay for the use you have made of it.

Probably not. I thought Ferrari was a common family name in Europe. But wanting a citation of that (which always seems to be required in this forum), I did a quick internet search.

Ferrari comes from ferraro, meaning “blacksmith,” putting it on par with the English & American surname “Smith.” And much like “Smith,” the Ferrari name is quite common — it’s the third most common surname in Italy.

https://www.ferrarilakeforest.com/ma...-ferrari-mean/

I don't think an overzealous corporate trademark quest would succeed in forcing you to remove this common word spray painted on the side of your car. Maybe in Europe - they sometimes have strange laws (like you can't call a cheese this or that unless it came from here or there). But in the US where I live, I really doubt that the car manufacturer of the same name would get any "usage" money out of me or force me to repaint my car. I don't think you can trademark a word either. You can trademark a logo that uses the word - the font, color, layout, graphic elements, etc. - but not the word itself. You may be able to trademark a non-word though (I'm thinking something like "Chick-fil-A" which is not a recognized real word - it's a made-up word).

But this is neither here nor there. I was just trying to create a silly analogy to putting "EBOK" in your computer files metadata. I highly doubt that Amazon has trademarked the pseudo-word "EBOK" in this context anyway.

[j.p.s]

Originally Posted by Sirtel

I could understand this if the time was wrong by, well, weeks or months. But others have experimented and found that deleting happens even when the time is wrong by a few minutes. Wouldn't that be "close enough"? I don't know, all this seems just weirder and weirder, and it's been going on for more than 2 years by now, as indicated by the posts in this thread. I'm not sure we'll ever see Amazon fixing it.

I have no idea how large the time error can be in this situation and be "safe", but I've personally encountered situations where one minute would be a big error, such as a network file server having the wrong time and a newly written file not being visible to a client for the duration of the error length because it was created "in the future".

Is it even certain that not using the EBOK tag prevents the book from being deleted?

No one outside (or maybe even inside) amazon is certain about the details. Message #530 implies this is the case, but there might be more than one cause for books to go missing. Some people report confirmation that book files are no longer on the kindle and others that the book files are there, but don't show up in the UI. (In addition some reports are factually inaccurrate.) So lots of murk surrounding this.

[DNSB]

Originally Posted by JSWolf

Actually, .azw is a generic files extension used by Amazon. It could be Mobi or KF8. It could have DRM or not.

Are we talking about the extension used when a file is downloaded by an app which seems to have no relationship to the ebook type (for instance, K4PC downloads all ebooks as .azw while K4A downloads everything as .prc.) or are we talking about the extensions used for different Amazon ebook formats? Where KF8/azw3, azw4, KFX, mobi, azw, etc. are used. In that use, .mobi is used for unprotected files and .azw is used for DRMmed files.

[DNSB]

For what it may be worth, when I was playing with this issue and EBOK and PDOC as cdetypes, I placed 8 books on my PW4 in pairs. 4 books had the cdetype in the EXTH header set to EBOK while 4 used PDOC as the cdetype (this was done using a hex editor to locate and edit the string). The 4 books with EBOK disappeared, the 4 books with PDOC remained when a WiFi connection to Amazon was done.

[j.p.s]

Thanks, that's the best and most comprehensive report so far. Was this a case where the book files were actually deleted or only disappeared from the UI? Also, did the .sdr directories remain or were they removed?