Indexing and thumbnails on Kindle Touch

[pirl8]

Some questions about kindke touch:

1. Book indexing on kindle touch

I'm using a Kindle Touch with 5.1.0 firmware. There's plenty of memory still available (>3.2Gb). I noticed that there were 8 books waiting to be indexed. After many hours they were still there. I removed all of them, added them back each at once, waiting for each to be indexed. All went well (read: everything indexed) till the last book. After that, my Kindle told me that 124 books had to be indexed. Can anybody explain me why? More precisely:

• How does indexing work on kindle touch?
• Is it possible to know which book is being indexed at a given time? How?
• Is it possible to know which percentage of a book has been indexed till now? How?
• Is it possible to force the indexing of a single book? How?
• Is it possible to enable/disable indexing? How?
• As far as I can see, indexes are saved to /mnt/us/system/Search Indexes. There's a single huge index.db file (maybe for the top level searching) and indexes relative to each book. The file names are 33 character strings. They look like MD5s but they all start with "y". Any clue how they are calculated?
• I usually find how many books have to be indexed by searching something. Is there a more direct way?

Maybe even with a partial knowledge of these points it could be possible to write an "indexing status" extension...

2. Book thumbnails on kindle touch

I've converted many ePubs into mobi with Calibre. A few of them - most noticeably some of the latest ones - don't get thumbnailed. The ePub validation seems OK and I didn't notice anything unusual on those titles...

Why some covers don't get thumbnailed?
Do they have to be a specific size or format?

I tried converting one the "offending" ePubs to mobi with latest Kindlegen (which packs mobi8+mobi6 in the same double sized file as far has I understand) instead of Calibre and it got thumbnailed. Could that be a calibre problem?

[knc1]

1. http://www.kabalatalisman.com/black-magic_protection/

2. Sounds like a format difference. Either a Kindlegen feature or a Calibre failure.

The ePub validation seems OK and I didn't notice anything unusual on those titles...

How did you do the examination to "notice" anything unusual? Visual? Software tools?

[geekmaster]

In our kindles, one protective talisman that shields (some of) us from seemingly arbitrary changes within closed source system software like the kindle framework, is direct framebuffer and /dev/input device access. The people who prefer "official" GUI framework code view low-level coders such as myself as computer anarchists, whilst the low-level coders view high-level coders as government cheese-eaters.

Personally, I do not want to forsake the framework completely -- I want code that uses and controls both methods, but these recent 5.1.0 changes attack both high-level AND low-level (unofficial) apps, via high-level changes like re-obfuscation, and the "Spanish device codes" breaking high-level custom apps, and low-level changes like making 5.1.x eink structures incompatible with 5.0.x eink structures without renaming them.

Perhaps amazon is taking some clues from Bill Gates here, in an effort to give preferential support to apps with their own official blessing (look up "Windows ain't done till Lotus won't run").

In my view, amazon's recent hacker-protection efforts have the real effect of making them less relevant, making me want to go to an even deeper level in the hardware (such as wanting my own loadable kernel module eink driver).

I gave the amazon tech's a "piece of my mind" on the 5.1.0 changes making it difficult to write apps that work on both 5.1.x and 5.0.x, and I requested that they release a new update that is compatible with 5.0.x so we can drop support for 5.1.0 and declare it "hopelessly broken". They said they will get back to me "in a couple of days". We shall see...

[knc1]

The situation does make a person want to replace the entire mess.

@GM: Have you had a chance to check into Mer?
http://merproject.org/
http://wiki.merproject.org/wiki/Main_Page

They are making arm v6l code releases:
http://releases.merproject.org/relea...mv6l/packages/
Those releases run on the Raspberry Pi:
http://www.bbc.com/news/technology-17190918
And others:
http://releases.merproject.org/releases/latest/builds/

They are making the entire project available as a "build system".
http://wiki.merproject.org/wiki/FakeOBS

I have not had time to do more than a quick read, not had time to try the builder.
Something I would want to do before even considering: "can it be ported to the Kindle display?" question.

Edit: 5/9/2012
I am taking a closer look at that project.
When they say: "large download" they mean it - about 7Gbyte to clone the repositories.

Edit: 5/10/2012
May well be an "all powerful" distribution builder but the learning curve is far too steep for this old man's mind.

[pirl8]

Originally Posted by geekmaster

The people who prefer "official" GUI framework code view low-level coders such as myself as computer anarchists

What about a stripped down version of a "dialog/xdialog" like tool? I guess you know it but, for those who don't, it's a command line linux tool to implement simple dialog boxes with applets like "yes-no" boxes, radio/check button lists, etc.
Many tasks could be easily implemented via shell scripts, if input could be given via a graphical interface...

In the meanwhyle, re-indexing is proceeding in my kindle. 25 books to the end. Indexing is mostly done in standby mode.

[NiLuJe]

@pirl8:

1/ It's not new, the framework has been known to just decide to re-parse everything on a whim, since at least as far back as FW 2.5 .

2/ Apparently, the KF/KT 5.1.0 doesn't like when the asin specified in EXTH header 113 isn't 'legit', and that seems to break KT thumbnails, or worse on the KF. (Check some calibre/kf threads for Kovid's findings on this). And calibre has been putting the uuid in there for quite some time (to support whispernet syncing, among other things), and that definitely doesn't look like an Amazon ASIN .

(Take that with a grain of salt, because I've had broken thumbnails with untouched azw/azw3 files fresh from Amazon's 'Transfer To Computer', too . [EDIT: But that might be because I've *sent* them via calibre, and AFAIR, it updates metadata on send even on DRM'ed books]).

[pirl8]

Originally Posted by NiLuJe

(Check some calibre/kf threads for Kovid's findings on this).

Thank you very much. I didn't even know that there was a Calibre section!!!
The thread is: here.

[ixtab]

Originally Posted by geekmaster

The people who prefer "official" GUI framework code view low-level coders such as myself as computer anarchists, whilst the low-level coders view high-level coders as government cheese-eaters.

Hehe... maybe a bit controversial, but definitely with a grain of truth. I tend to think of myself as somewhere in between: not really afraid of low-level stuff, but not really interested in very high-level stuff either. The "sweet spot" for me is somewhere in between: where much of the high-level stuff can be altered, but (if possible) without having to deal with all the gory low-level details.

Originally Posted by geekmaster

Personally, I do not want to forsake the framework completely -- I want code that uses and controls both methods, but these recent 5.1.0 changes attack both high-level AND low-level (unofficial) apps, via high-level changes like re-obfuscation, and the "Spanish device codes" breaking high-level custom apps, and low-level changes like making 5.1.x eink structures incompatible with 5.0.x eink structures without renaming them.

Perhaps amazon is taking some clues from Bill Gates here, in an effort to give preferential support to apps with their own official blessing (look up "Windows ain't done till Lotus won't run").

Amen. I can't help it, but I'm feeling like 5.1.0 was indeed a huge step backwards (despite the "customer-friendly" additions like KF8, WPA-EAP and "official" localization). Then again, they changed so many things for the worse, I don't even know where to start.

Originally Posted by geekmaster

I gave the amazon tech's a "piece of my mind" on the 5.1.0 changes making it difficult to write apps that work on both 5.1.x and 5.0.x, and I requested that they release a new update that is compatible with 5.0.x so we can drop support for 5.1.0 and declare it "hopelessly broken". They said they will get back to me "in a couple of days". We shall see...

Don't hold your breath. I guess it's like knc1 wrote in another thread: "Not invented here, so go screw yourselves" (paraphrasing). I think we (unfortunately) ended up in the classical "arms race", where we are perceived as the bad, evil, hackers, and the company tries to do everything from preventing us to tamper with their holy device. What a short-sighted and stupid prejudice -- trying to prevent their own device from being leveraged to its full potential. Ahh whatever... some people just don't WANT to understand.

[knc1]

Originally Posted by ixtab

Amen. I can't help it, but I'm feeling like 5.1.0 was indeed a huge step backwards (despite the "customer-friendly" additions like KF8, WPA-EAP and "official" localization). Then again, they changed so many things for the worse, I don't even know where to start.
- - - -
Don't hold your breath. I guess it's like knc1 wrote in another thread: "Not invented here, so go screw yourselves" (paraphrasing).
- - - -

I think that the Amazon's development plan for v-5.1 was leaked about six years ago:
http://www.jir.com/graph_contest/index.html#OneGraph
Where v-5.1 was codenamed: "16"

You can almost see how Amazon has been trying to "Normalize" the "Kindle Experience" in the following article and the article's bar graph:
http://www.mediacosm.com/eating/

Note: JIR - Journal of Irreproducible Results
lab126 is expected to publish any day, RSN.

[geekmaster]

Every day I receive one or two emails from the tech at amazon asking me to click a "yes you solved my problem" or a "no you did not solve my problem" link. There is no third option to select, and no way to provide extra details. I am waiting for that contact he said I would get "in a couple of days" before I give him my feedback.

It has been long enough that I am tempted to click the "no help" link...

[knc1]

Originally Posted by geekmaster

Every day I receive one or two emails from the tech at amazon asking me to click a "yes you solved my problem" or a "no you did not solve my problem" link. There is no third option to select, and no way to provide extra details. I am waiting for that contact he said I would get "in a couple of days" before I give him my feedback.

It has been long enough that I am tempted to click the "no help" link...

Quote them a high price for your "Consult" - That might at least get you a "go to hell" from accounts payable.

[knc1]

Not sure just how we got this thread so far off topic, but
at least the O.P. seems to have gotten his books to index:

Originally Posted by pirl8

In the meanwhyle, re-indexing is proceeding in my kindle. 25 books to the end. Indexing is mostly done in standby mode.

An occasional running thread of humor and comments on "life as it exists at Amazon" can only add a bit of spice to the forum and shouldn't do any harm.

[pirl8]

Originally Posted by knc1

Not sure just how we got this thread so far off topic, but at least the O.P. seems to have gotten his books to index:

Actually I don't know what has been indexed. I tried to open the book indexed last (which incidentally was one of the first ebooks i've ever scanned → OCR → ODT → ePub → Sigil/Notepad++ → Calibre → mobi → put on my kindle) and - SURPRISE - it had actually not been indexed at all.

[knc1]

Originally Posted by pirl8

Actually I don't know what has been indexed. I tried to open the book indexed last (which incidentally was one of the first ebooks i've ever scanned → OCR → ODT → ePub → Sigil/Notepad++ → Calibre → mobi → put on my kindle) and - SURPRISE - it had actually not been indexed at all.

Still have the ODT?
Does Kindle index PDF?
Maybe try: OCR -> ODT -> PDF

Or some conversion chain in an attempt to avoid the Calibre -> mobi conversion step.
Or maybe try running it through the Amazon "Free (wifi), convert by e-mail" service.

Anything at all that might help give an idea of where in that long conversion chain things go wrong.

- - - -

Yes, we are paying attention to your topic. Just don't have any ideas about the source of the problem other than the eariler post that points out that sometimes even Amazon books don't index.

[pirl8]

Originally Posted by knc1

Just don't have any ideas about the source of the problem other than the eariler post that points out that sometimes even Amazon books don't index.

I know this and I wasn't indeed asking a solution. Just wondering if anyone has a clue on how indexing gets done.

The funny thing is that I'm quite sure that the same identical mobi had been correctly indexed before. Right now I backupped the mobi, deleted it from the Kindle via GUI, manually removed the related directories, added it back (via simple file copy) and - EVEN MORE SURPRISE - now it has been properly indexed.

It seems to me that there are two levels of indexing: one to search the single book, and one to search the whole library. If one of these fails for any reason the kindle gives up indexing thus leaving the database incomplete maybe in the hope of having a better luck next time. A fatalistic algorithm?