LibrarianSync: Kindle collections builder

[barsanuphe]

What it is.

LibrarianSync is a Kindle collection builder, originally meant as a complement to Librarian, but which can be used completely independently.

It has been tested on a Kindle PW2 (EU, Wi-Fi) FW 5.4.5.1, and on a Touch FW 5.3. It probably works on other K5 devices. It apparently works on the Voyage as well.

It can use as a source:

• a json file in native format (such as generated by Librarian when syncing, see below)
• the json file automatically generated by the Calibre Kindle Collections plugin)
• the folder structure inside the documents/ folder.

Using the json file, the entries can either remplace all existing collections or update existing collections. Using folder structure, all previously existing collections are dropped.

The native json format looks as follows:

{
  "library/Alexandre Dumas/Alexandre Dumas (2004) Les Trois Mousquetaires.mobi": ["gutenberg","french","not read"],
  "library/Alexandre Dumas/Alexandre Dumas (2011) Le Comte De Monte-Cristo.mobi": ["gutenberg","french","most awesome book ever"],
  "re:Alexandre Dumas (Père|Fils)": ["dumas"]
}

As shown in the last line, you can add (python) regular expressions instead of filenames (with a re: prefix); all matches will be added to the associated collection.

What is needed.

LibrarianSync comes as a KUAL menu entry. As such, it requires: a jailbroken PW2, MKK, KUAL, and python (snapshot > 0.10N-r10867).

For more information, see the README file on github.

Where it is.

The code resides on github, but since I do not use this personally anymore, the better place to look for changes is NiLuJe's fork (thanks!).
See attachments for an up to date Kindle package.

Note: the code has moved to a separate repository as LibrarianSync has become something useful in its own right. Please update your git remotes to the new repository.

Where it comes from.

LibrarianSync is inspired by this thread.

Thanks to NiLuJe for all the contributions and continuous support, and kaznelson for his initial courageous sqlite investigations.

Feedback is very much welcome.

LibrarianSync Changelog

v20201109 - Same as below . -- NiLuJe
v20190103 - Refreshed package from latest sources, so as not to leave four year old packages around... -- NiLuJe
v1.10 - fixed when getting a null cdetype, added Voyage support.
v1.9 - fixed support of non-cloud aware devices
v1.8 - fixed collection appearing twice in FW 5.6
v1.7 - added support for regular expressions in librarian json when building collections with LibrarianSync.
v1.6 - fixed exporting collections, added downloading ebooks hosted over http by librarian.
v1.5 - package available for all k5 models.
v1.4 - updating collections is now faster.
v1.3 - calibre plugin json support & ability to export current collections (librarian and calibre plugin versions). While running the script, current operation now logged and displayed.
v1.2 - experimental calibre plugin json support & ability to export current collections to extensions/exported_collections.json.
v1.1 - experimental calibre plugin json support
v1.0 - original version

[knc1]

Originally Posted by barsanuphe

Thanks to all the (old but still relevant) information in this thread I managed to write a python script to automatically generate collections on my pw2.
Instead of looking at the files or directory structure, it uses information in a json file to create and populate collections. This way it's possible to manage collections just with one simple text file, such as:



In case someone might find this helpful, the script and KUAL menu entry can be found at https://github.com/barsanuphe/librar.../librariansync.

Thanks for this.

It comes at a good time, since the interactive Collection Manager is not (yet) ported to all of the PW2 firmware / hardware combinations.

Note:
I think this will also work on other models of Kindles - that DB probably doesn't (didn't) change that often.

[barsanuphe]

Yes, I guess it should work with other models, after all this thread was originally about the Kindle Touch. I only have a pw2 (EU), so I can't verify that. If anyone is willing to try it on a different model, feedback would be welcome.

Also, I updated the script and KUAL menu to allow for incremental collection updates, that do not rebuild all collections but merely add the json entries into existing or new collections.

[NiLuJe]

@barsanuphe: I'll probably get my ass in gear and bundle the sqlite3 & requests Python modules in the next snapshots, FYI; that should hopefully let you get rid of the curl & sqlite3 forks.

EDIT: Done .

[knc1]

The Amazon Lua install includes the sqlite and json add-on libraries as I recall (at least on the PW1, I think on the KT also, PW2 is unknown).

If someone wanted to get ambitious, they could re-write this in Lua and avoid any add-in support packages.

The same code (and libraries) shipped by Amazon can also be run by the ARMhf Luajit if the user had those installed.

- - - -

Just a thought -
Not something I have time to do.
And twobob is still off, dealing with "life".

[barsanuphe]

Originally Posted by NiLuJe

@barsanuphe: I'll probably get my ass in gear and bundle the sqlite3 & requests Python modules in the next snapshots, FYI; that should hopefully let you get rid of the curl & sqlite3 forks.

EDIT: Done .

Great! I hadn't seen your update. I updated my script to use the new modules.
I didn't know the requests module, it's pretty nifty.

[NiLuJe]

@barsanuphe: That was fast ^^.

Yep, requests is great . Did the switch affect performance in any measurable way?

[kaznelson]

barsanuphe
Whether it's possible to make "lite" version?
To create collections on folders names (without metainfo), and worked ONLY on Kindle? (without computer)

[barsanuphe]

@NiLuJe:

I have not looked at the performance aspect yet, and am testing with only a few ebooks on my Kindle. I expect it to be faster though, but couldn't tell you by how much. Also, all operations are now sent in a single POST request, which should help too.

@kaznelson:

I updated the script and added a new entry in the KUAL menu to that effect. You should now be able to create collections from your folder structure, without json file. The collection name is the path of each ebook (without filename), relative to /mnt/us/documents.

ie if you have "/mnt/us/documents/sf/franck herbert/dune/dune.epub", the collection name will be "sf/franck herbert/dune".

I tried earlier to make nested collections, but I found only top collections are displayed on my Kindle. A subcollection can be added to a collection but does not show up on the interface. Reading through the Collection Manager thread, it seems it's a firmware issue...

[NiLuJe]

@barsanuphe: Yup, nesting was broken in FW 5.4.2, and it hasn't resurfaced since.

I was thinking of using this with the Calibre Kindle Collections plugin, most likely by making it understand the json schema used by the plugin, since the plugin already takes care of a number of sanity checks (namely the uuid/ASIN handling of each book).

FWIW, the schema is as follows:

{
	"<CollectionName>@<Language>":
	{
		"items":
		[
			"<hash>"
		],
		"lastAccess": <epoch>
	}
}

So, as an example with a few different types of hash, that would look something like:

{
	"Apps@en-US":
	{
		"items":
		[
			"#B006VZUZV2^AZW2"
		],
		"lastAccess": 1344387337000
	},
	"Brandon Sanderson@en-US":
	{
		"items":
		[
			 "*c98b6567481d950dccad195379053e2539f1d75c",
			 "#12e45a36-0410-4caf-b32d-dc807713d8e9^EBOK"
		],
		"lastAccess": 1234567889976
	}
}

(The indentation is all mine, for formatting purposes only).

[kaznelson]

barsanuphe
Thanks, it work.

Two small adding:

1) File generate_collections.py, line 53: [".epub", ".mobi", ".azw3"]
Amazon books standard formats: [".azw", ".azw3", ".mobi", ".prc", ".pdf", ".txt"]

2) Books from /mnt/us/documents/ are added in collection "root" - is incorrect, imho.

[barsanuphe]

@NiLuJe
I'm not familiar with the calibre plugin, where do the items hashes come from? Are they generated by the plugin (which means they would have to be created on the kindle) or are they extracted from the Kindle db?

@kaznelson
1) updated with your list
2) books in documents/ are now ignored. I agree the root collection made no sense

[NiLuJe]

@barsanuphe: It's in fact the original format used by the Collections feature on Kindles before FW 5.x came along .

The hash is generated (in this instance, by the plugin) from the metadata of the file, it follows the format '#ASIN^cdeType' if an ASIN is set [valid, like in my Apps exemple, or invalid [Calibre generated], like in the second collection. If the ASIN isn't set, it's formatted as '*hash', where hash is the md5 of the absolute path of the file (using /mnt/us as the userstore root).
As for the cdeType, I *think* it's mandatory for a file to be valid/collectionable, and if it isn't, I'm assuming that defaults to EBOK or PDOC.
Anyway, the plugin handles that aspect to make sure that what's end up in the json is sane.

On FW 5.x, in the SQLite db, those are stored in two distinct fields, as p_cdeKey & p_cdeType in the Entries table, and i_member_cde_type & i_member_cde_key in the Collections table, with the notable difference that the '#' is *NOT* present anymore in front of the ASIN (the '*' of the legacy hashes stays, though).

TL;DR: You can get the db uuid of an item from the db w/ these .

[kaznelson]

Experiments with collections (excuse for my english ).

To collections created with LibSync, it's impossible to standardly add books without problems.
Possible to open "Add to Collections" screen, but after a book "check" and press "Done" - my Kindle don't exit on "Home" screen! Only reboot.
Also automatically added books don't hide from "Home" screen, if are added in a LibSync collections.

I compared cc.db, created standardly offline and LibSync:
p_type "Collection" (collections)

Original file cc.db:				LibSync file cc.db:
p_isArchived = 1				p_isArchived = 0
j_collections = NULL				j_collections = []
p_collectionCount = NULL			p_collectionCount = 0
p_collectionDataSetName = <p_uuid from cc.db>	p_collectionDataSetName = 0

p_type "Entry:Item" (books)

Original file cc.db:			LibSync file cc.db:
p_collectionCount = 1			p_collectionCount = 0

Also LibSync don't fill a file wsync.db. At standard offline creation of a collection with name "Misc":

In table DataSetProperties string is added

CloudCollections.Items<p_uuid from cc.db>	<p_uuid from cc.db>	CloudCollections.Items	<...and more...>

In table SyncRecords string is added

CloudCollectionsCollectionEntries<p_uuid from cc.db>	CloudCollectionsCollectionEntries	<p_uuid from cc.db>	"{""type"":""BOOKS"",""title"":""Misc"",""sortTitle"":""Misc"",""phoneticTitle"":""Misc"",""language"":""en-GB"",""version"":""1.0""}"	<...and more...>

________________________________________

In LibSync collections +wsync.db i made these changes manually (by SQLiteStudio). After reboot - errors disappeared.

I don't know, all is necessary or not. But possibly it's necessary to fill wsync.db and to specify uuid collections in a column p_collectionDataSetName (cc.db).

And probably, collection locale should take from system locale (en-US or...)

[NiLuJe]

@kaznelson: That vaguely reminds me of a similar user report w/ CM... Does CM do it 'properly'?