![]() |
#1 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
LibrarianSync: Kindle collections builder
What it is.
LibrarianSync is a Kindle collection builder, originally meant as a complement to Librarian, but which can be used completely independently. It has been tested on a Kindle PW2 (EU, Wi-Fi) FW 5.4.5.1, and on a Touch FW 5.3. It probably works on other K5 devices. It apparently works on the Voyage as well. It can use as a source:
Using the json file, the entries can either remplace all existing collections or update existing collections. Using folder structure, all previously existing collections are dropped. The native json format looks as follows: Code:
{ "library/Alexandre Dumas/Alexandre Dumas (2004) Les Trois Mousquetaires.mobi": ["gutenberg","french","not read"], "library/Alexandre Dumas/Alexandre Dumas (2011) Le Comte De Monte-Cristo.mobi": ["gutenberg","french","most awesome book ever"], "re:Alexandre Dumas (Père|Fils)": ["dumas"] } What is needed. LibrarianSync comes as a KUAL menu entry. As such, it requires: a jailbroken PW2, MKK, KUAL, and python (snapshot > 0.10N-r10867). For more information, see the README file on github. Where it is. The code resides on github, but since I do not use this personally anymore, the better place to look for changes is NiLuJe's fork (thanks!). See attachments for an up to date Kindle package. Note: the code has moved to a separate repository as LibrarianSync has become something useful in its own right. Please update your git remotes to the new repository. Where it comes from. LibrarianSync is inspired by this thread. Thanks to NiLuJe for all the contributions and continuous support, and kaznelson for his initial courageous sqlite investigations. Feedback is very much welcome. LibrarianSync Changelog v20201109 - Same as below ![]() v20190103 - Refreshed package from latest sources, so as not to leave four year old packages around... -- NiLuJe v1.10 - fixed when getting a null cdetype, added Voyage support. v1.9 - fixed support of non-cloud aware devices v1.8 - fixed collection appearing twice in FW 5.6 v1.7 - added support for regular expressions in librarian json when building collections with LibrarianSync. v1.6 - fixed exporting collections, added downloading ebooks hosted over http by librarian. v1.5 - package available for all k5 models. v1.4 - updating collections is now faster. v1.3 - calibre plugin json support & ability to export current collections (librarian and calibre plugin versions). While running the script, current operation now logged and displayed. v1.2 - experimental calibre plugin json support & ability to export current collections to extensions/exported_collections.json. v1.1 - experimental calibre plugin json support v1.0 - original version Last edited by barsanuphe; 11-21-2021 at 07:22 PM. Reason: added link to the active fork |
![]() |
![]() |
![]() |
#2 | |
Going Viral
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 17,212
Karma: 18210809
Join Date: Feb 2012
Location: Central Texas
Device: No K1, PW2, KV, KOA
|
Quote:
It comes at a good time, since the interactive Collection Manager is not (yet) ported to all of the PW2 firmware / hardware combinations. Note: I think this will also work on other models of Kindles - that DB probably doesn't (didn't) change that often. |
|
![]() |
![]() |
Advert | |
|
![]() |
#3 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
Yes, I guess it should work with other models, after all this thread was originally about the Kindle Touch. I only have a pw2 (EU), so I can't verify that. If anyone is willing to try it on a different model, feedback would be welcome.
Also, I updated the script and KUAL menu to allow for incremental collection updates, that do not rebuild all collections but merely add the json entries into existing or new collections. |
![]() |
![]() |
![]() |
#4 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@barsanuphe: I'll probably get my ass in gear and bundle the sqlite3 & requests Python modules in the next snapshots, FYI; that should hopefully let you get rid of the curl & sqlite3 forks.
EDIT: Done ![]() Last edited by NiLuJe; 08-26-2014 at 08:10 PM. |
![]() |
![]() |
![]() |
#5 |
Going Viral
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 17,212
Karma: 18210809
Join Date: Feb 2012
Location: Central Texas
Device: No K1, PW2, KV, KOA
|
The Amazon Lua install includes the sqlite and json add-on libraries as I recall (at least on the PW1, I think on the KT also, PW2 is unknown).
If someone wanted to get ambitious, they could re-write this in Lua and avoid any add-in support packages. The same code (and libraries) shipped by Amazon can also be run by the ARMhf Luajit if the user had those installed. - - - - Just a thought - Not something I have time to do. And twobob is still off, dealing with "life". |
![]() |
![]() |
Advert | |
|
![]() |
#6 | |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
Quote:
I didn't know the requests module, it's pretty nifty. |
|
![]() |
![]() |
![]() |
#7 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@barsanuphe: That was fast ^^.
Yep, requests is great ![]() |
![]() |
![]() |
![]() |
#8 |
Old Kaz
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 208
Karma: 1064151
Join Date: May 2010
Device: PocketBook Era
|
barsanuphe
Whether it's possible to make "lite" version? To create collections on folders names (without metainfo), and worked ONLY on Kindle? (without computer) |
![]() |
![]() |
![]() |
#9 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
@NiLuJe:
I have not looked at the performance aspect yet, and am testing with only a few ebooks on my Kindle. I expect it to be faster though, but couldn't tell you by how much. Also, all operations are now sent in a single POST request, which should help too. @kaznelson: I updated the script and added a new entry in the KUAL menu to that effect. You should now be able to create collections from your folder structure, without json file. The collection name is the path of each ebook (without filename), relative to /mnt/us/documents. ie if you have "/mnt/us/documents/sf/franck herbert/dune/dune.epub", the collection name will be "sf/franck herbert/dune". I tried earlier to make nested collections, but I found only top collections are displayed on my Kindle. A subcollection can be added to a collection but does not show up on the interface. Reading through the Collection Manager thread, it seems it's a firmware issue... |
![]() |
![]() |
![]() |
#10 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@barsanuphe: Yup, nesting was broken in FW 5.4.2, and it hasn't resurfaced since.
I was thinking of using this with the Calibre Kindle Collections plugin, most likely by making it understand the json schema used by the plugin, since the plugin already takes care of a number of sanity checks (namely the uuid/ASIN handling of each book). FWIW, the schema is as follows: Code:
{ "<CollectionName>@<Language>": { "items": [ "<hash>" ], "lastAccess": <epoch> } } Code:
{ "Apps@en-US": { "items": [ "#B006VZUZV2^AZW2" ], "lastAccess": 1344387337000 }, "Brandon Sanderson@en-US": { "items": [ "*c98b6567481d950dccad195379053e2539f1d75c", "#12e45a36-0410-4caf-b32d-dc807713d8e9^EBOK" ], "lastAccess": 1234567889976 } } Last edited by NiLuJe; 08-29-2014 at 07:01 PM. |
![]() |
![]() |
![]() |
#11 |
Old Kaz
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 208
Karma: 1064151
Join Date: May 2010
Device: PocketBook Era
|
barsanuphe
Thanks, it work. ![]() Two small adding: 1) File generate_collections.py, line 53: [".epub", ".mobi", ".azw3"] Amazon books standard formats: [".azw", ".azw3", ".mobi", ".prc", ".pdf", ".txt"] 2) Books from /mnt/us/documents/ are added in collection "root" - is incorrect, imho. |
![]() |
![]() |
![]() |
#12 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
@NiLuJe
I'm not familiar with the calibre plugin, where do the items hashes come from? Are they generated by the plugin (which means they would have to be created on the kindle) or are they extracted from the Kindle db? @kaznelson 1) updated with your list 2) books in documents/ are now ignored. I agree the root collection made no sense ![]() |
![]() |
![]() |
![]() |
#13 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@barsanuphe: It's in fact the original format used by the Collections feature on Kindles before FW 5.x came along
![]() The hash is generated (in this instance, by the plugin) from the metadata of the file, it follows the format '#ASIN^cdeType' if an ASIN is set [valid, like in my Apps exemple, or invalid [Calibre generated], like in the second collection. If the ASIN isn't set, it's formatted as '*hash', where hash is the md5 of the absolute path of the file (using /mnt/us as the userstore root). As for the cdeType, I *think* it's mandatory for a file to be valid/collectionable, and if it isn't, I'm assuming that defaults to EBOK or PDOC. Anyway, the plugin handles that aspect to make sure that what's end up in the json is sane. On FW 5.x, in the SQLite db, those are stored in two distinct fields, as p_cdeKey & p_cdeType in the Entries table, and i_member_cde_type & i_member_cde_key in the Collections table, with the notable difference that the '#' is *NOT* present anymore in front of the ASIN (the '*' of the legacy hashes stays, though). TL;DR: You can get the db uuid of an item from the db w/ these ![]() Last edited by NiLuJe; 08-29-2014 at 07:06 PM. |
![]() |
![]() |
![]() |
#14 |
Old Kaz
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 208
Karma: 1064151
Join Date: May 2010
Device: PocketBook Era
|
Experiments with collections (excuse for my english
![]() To collections created with LibSync, it's impossible to standardly add books without problems. Possible to open "Add to Collections" screen, but after a book "check" and press "Done" - my Kindle don't exit on "Home" screen! Only reboot. Also automatically added books don't hide from "Home" screen, if are added in a LibSync collections. I compared cc.db, created standardly offline and LibSync: p_type "Collection" (collections) Code:
Original file cc.db: LibSync file cc.db: p_isArchived = 1 p_isArchived = 0 j_collections = NULL j_collections = [] p_collectionCount = NULL p_collectionCount = 0 p_collectionDataSetName = <p_uuid from cc.db> p_collectionDataSetName = 0 Code:
Original file cc.db: LibSync file cc.db: p_collectionCount = 1 p_collectionCount = 0 In table DataSetProperties string is added Code:
CloudCollections.Items<p_uuid from cc.db> <p_uuid from cc.db> CloudCollections.Items <...and more...> Code:
CloudCollectionsCollectionEntries<p_uuid from cc.db> CloudCollectionsCollectionEntries <p_uuid from cc.db> "{""type"":""BOOKS"",""title"":""Misc"",""sortTitle"":""Misc"",""phoneticTitle"":""Misc"",""language"":""en-GB"",""version"":""1.0""}" <...and more...> In LibSync collections +wsync.db i made these changes manually (by SQLiteStudio). After reboot - errors disappeared. I don't know, all is necessary or not. But possibly it's necessary to fill wsync.db and to specify uuid collections in a column p_collectionDataSetName (cc.db). And probably, collection locale should take from system locale (en-US or...) Last edited by kaznelson; 08-30-2014 at 09:37 AM. |
![]() |
![]() |
![]() |
#15 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@kaznelson: That vaguely reminds me of a similar user report w/ CM... Does CM do it 'properly'?
Last edited by NiLuJe; 08-30-2014 at 09:50 AM. |
![]() |
![]() |
![]() |
Tags |
kindle collections, librariansync |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Vocabulary builder and flash card feature on old kindle 2/3/dx? | dark.knight1 | Kindle Developer's Corner | 12 | 10-10-2015 01:12 PM |
Vocabulary builder | bobafruit | Kindle Developer's Corner | 25 | 09-04-2015 01:46 AM |
Free (nook/Kindle/iTunes/DRM-free) Legacy Builder [Xtian Business Leadership Advice] | ATDrake | Deals and Resources (No Self-Promotion or Affiliate Links) | 0 | 10-13-2014 02:44 AM |
Vocabulary Builder for Kindle Paperwhite 1 | pxpxpx | Amazon Kindle | 2 | 02-16-2014 06:07 PM |
SBPubX Builder | tmaynard | Fictionwise eBookwise | 16 | 11-20-2008 12:35 PM |