![]() |
#1 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,658
Karma: 85000001
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
|
KRDS - A parser for Kindle reader data store files
A recent discussion prompted me to look into how annotations are stored on Kindle devices running recent firmware versions.
Information related to each book being read is saved in a pair of sidecar files in the book's .sdr folder. These files contain serialized data objects used by the e-book reader application. The first file contains objects that change with every page turn such as the last page read and reading timing. The second file contains less frequently changed data such as personal annotations, font & dictionary choices, and synced reading position. The file extensions used depend on the book format:
The data format appears to be proprietary to Amazon and is similar to the Amazon Ion Binary Encoding used by KFX. It encodes the name of each object being serialized along with a list of property values. Values each have an associated data type, such as integer or string. Decoding objects requires knowledge of the data structure associated with each class. KRDS (Kindle Reader Data Store) I have written a Python script to parse these files. The main function accepts an input file name, parses it into a Python data structure, and outputs the result as a human readable JSON file. I reverse engineered the data structures for several classes commonly used by the Kindle reader, but it is likely that I missed some things. Reports of any file that is not handled properly are welcome. For the latest version see this post by tomsem. Usage Spoiler:
Sample Output Spoiler:
Last edited by jhowell; 06-10-2023 at 07:04 AM. Reason: Link to new version by tomsem |
![]() |
![]() |
![]() |
#2 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 5,416
Karma: 99236514
Join Date: Apr 2011
Device: pb360
|
Wow! Thanks!
I'll try kicking the tires when I get some time. |
![]() |
![]() |
Advert | |
|
![]() |
#3 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,658
Karma: 85000001
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
|
|
![]() |
![]() |
![]() |
#4 | |
curly᷂͓̫̙᷊̥̮̾ͯͤͭͬͦͨ ʎʌɹnɔ
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,007
Karma: 50506927
Join Date: Dec 2010
Location: ♁ ᴺ₄₅°₃₀' ᵂ₇₃°₃₇' ±₆₀"
Device: K3₃.₄.₃ PW3&4₅.₁₃.₃
|
Quote:
Spoiler:
Last edited by PoP; 08-13-2019 at 04:17 PM. Reason: add generated json |
|
![]() |
![]() |
![]() |
#5 | |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,658
Karma: 85000001
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
|
Quote:
I tested a Topaz (.azw1) file and it uses .tal and .tas files with the same type of content. I will update the first post to add this information. |
|
![]() |
![]() |
Advert | |
|
![]() |
#6 |
Guru
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 960
Karma: 10500008
Join Date: Feb 2010
Location: Serbia
Device: Kindle PW5, Kobo Libra 2, Kindle PW1
|
So, this script does not extract the actual text that was highlighted?
|
![]() |
![]() |
![]() |
#7 | |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,658
Karma: 85000001
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
|
Quote:
The linkage between the files that this program decodes and the book's content are fields labeled with "position" in the name. These are strings that identify where to find content within a book and are interpreted differently for each book format. KF8 (azw3) format appears to be the simplest case. The position is a decimal number giving an offset within the raw HTML content of the book, as can be obtained using the kindleunpack software. See the work done by j.p.s for an example of how to make use of this information. MOBI (azw) format is similar, but there appears to be additional information that I have not attempted to decode. KFX uses two values separated by a colon. The first is a base64 encoding of the eid and offset, which are fields used internally by KFX to determine the location of content. The second is the actual position number, which in the case of KFX counts visible unicode characters instead of raw HTML bytes. I have not looked into how position numbers are handled in the other formats that Kindle supports. Last edited by jhowell; 08-14-2019 at 08:16 AM. |
|
![]() |
![]() |
![]() |
#8 |
Junior Member
![]() Posts: 8
Karma: 10
Join Date: Dec 2015
Device: Kindle Voyage
|
Hi could you please make a python version of the script that extract the actual text that was highlighted? j.p.s's code is in c which is really difficult for me (and most people without sufficent programming knowledges I guess) to use. Thanks.
|
![]() |
![]() |
![]() |
#9 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 5,416
Karma: 99236514
Join Date: Apr 2011
Device: pb360
|
Have you tried the perl version azw3r.pl? That doesn't require compiling and also somewhat obsoletes the C version since it is more robust.
|
![]() |
![]() |
![]() |
#10 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,658
Karma: 85000001
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
|
|
![]() |
![]() |
![]() |
#11 | |
Junior Member
![]() Posts: 8
Karma: 10
Join Date: Dec 2015
Device: Kindle Voyage
|
Quote:
If anyone else has the same needs: I used johwell's script to get the highlight json file, then used it as an index to extract highlight text from the "My Clippings.txt" file by matching the time of creation of the highlight (there seems to be 1-2 seconds offset in some cases). Since "My Clippings.txt" is guaranteed a superset of the azw3r file, it worked out perfectly! (The original reason that I don't use "My Clippings.txt" directly is that it contains redundant highlights, which for example were deleted from the book.) Last edited by paulat; 12-03-2019 at 09:03 AM. |
|
![]() |
![]() |
![]() |
#12 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,658
Karma: 85000001
Join Date: Nov 2011
Location: Charlottesville, VA
Device: Kindles
|
|
![]() |
![]() |
![]() |
#13 | |||
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 5,416
Karma: 99236514
Join Date: Apr 2011
Device: pb360
|
Quote:
Quote:
Quote:
I also haven't used my highlight tools for a while and had forgotten about krdsJSON2notes.pl which uses the output of jhowell's krds.py to extract highlights from the book text. krdsJSON2notes.pl is in azw3r-0.17.zip is attached to post #1 in my highlight extraction thread. |
|||
![]() |
![]() |
![]() |
#14 | |
livin' with ebooks
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 36
Karma: 42048
Join Date: Jun 2014
Location: Berlin (DE), winters in Aomori (JP), from Montréal (CA)
Device: Libra2, KOA
|
Quote:
![]() |
|
![]() |
![]() |
![]() |
#15 |
Zealot
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 136
Karma: 493152
Join Date: Mar 2012
Location: Spain
Device: Kindle Oasis 2
|
Very interesting functionality. I am doing a bit of research on the data that comes out and the most useful thing I have achieved is the following that I explain with an example:
Code:
{ "timer.model": { "version": 3, "totalTime": 35191528, "totalWords": 131355, "totalPercent": 1.0758328462887201, "averageCalculator": { "samples1": [], "samples2": [], "normalDistributions": [ { "count": 466, "sum": 102178.8613209594, "sumOfSquares": 23794574.149871096 }, { "count": 17, "sum": 9752.422438256466, "sumOfSquares": 5781216.939200989 } ], "outliers": [ [ 52.47694415740492 ], [ 833.0194263574235 ], [ 389.67816758480717, 395.84946569614374, 416.29497472494796 ] ] } }, "fpr": { "position": "997716", "time": null, "timeZoneOffset": null, "country": "", "device": "" }, "book.info.store": { "numberOfWords": 167321, "percentOfBook": 1.3649912331969611 }, "page.history.store": [ { "position": "77096", "time": "2021-04-18T23:56:15.675000" }, { "position": "169022", "time": "2021-04-21T17:47:22.352000" }, { "position": "836132", "time": "2021-05-04T00:19:54.125000" } ], "lpr": { "position": "997716", "time": "2021-05-07T21:49:36.114000" }, "whisperstore.migration.status": [ false, false ] } totalTime is the total read time in milliseconds. Once approximately fifteen samples have been made (each sample is taken when the page is turned), statistics come out. Under normalDistributions comes out count which refers to the samples taken. sum refers to the sum of words per minute. With which if we divide sum / count we get the average reading in words per minute. In the example, 102 178/466 = 219 outliers are the samples that deviate from the mean. This usually happens at the beginning of the book when we skip a lot of useless words before the actual beginning of the book. Of the various normalDistributions, the first is composed of normal page reads in which after a few tests, values of pages are entered which have been read at a speed of between 0 and 800 words per minute approximately. The second consists of readings with more than 800 words per minute. This happens when we go back and forth in the book just to look at something that we may have forgotten. This is not a normal reading and gives some anomalous data of words per minute with which they go to the other collection so that they do not distort the real data. If the book is left open without locking, it counts the reading speed until it locks. Therefore, to make the calculation better, when we are not reading, it is best to lock the device. If we are sleepy and fall asleep, too bad ... that counts as slow reading (and it's true, ha ha ha) Last edited by Shark69; 07-29-2021 at 01:47 PM. |
![]() |
![]() |
![]() |
Tags |
krds |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Where does Reader for PC store files? | redbees | Sony Reader | 13 | 01-22-2021 09:07 AM |
Lost Kindle Data files | Alastair_Lack | Amazon Kindle | 5 | 06-28-2013 03:31 PM |
Where does the Android Kindle app store ebook files? | bfollowell | Amazon Kindle | 3 | 03-12-2013 05:04 PM |
Where does Connect Reader store ebook files | 1490peter | Sony Reader | 1 | 08-12-2009 03:45 AM |
Old RSS data on Sony Store | JeffASonyReader | Sony Reader | 3 | 01-14-2008 06:20 PM |