LibrarianSync: Kindle collections builder

[NiLuJe]

Okay, I moved the relevant posts from the old thread, and merged some stuff to somewhat reorder it properly.

Give me a shout if I broke something .

[barsanuphe]

thanks NiLuJe, everything looks good.

BTW the new export option for LibrarianSync uses my own json format, and not the format used by the calibre kindle plugin. Would that be useful or even make sense?

[NiLuJe]

@barsanuphe: Both CM and the Calibre Plugin only understand the legacy json schema, so, yeah, unless you also think of something to do with an export inside LibrarianSync itself, yeah it'd probably be more useful in the other format .

(Granted, the interaction with CM might not be terribly useful, but the Calibre Plugin features a GUI to manually manage the collections, which some people find useful).

I haven't yet checked how it behaves, but, FWIW, the exact filename CM & the Plugin expect is [/mnt/us/]system/collections.json .

[NiLuJe]

Okay, testing on a Touch, kablooey!

140907:023456 cvm[1922]: E rc:lwsf:http_code=0,handler=process_/change,luaerr=LUA_ERRRUN(2):worker script failed: /usr/lib/ccat/change.lua:1371: /usr/lib/ccat/change.lua:1055: Attempt to set unknown field collectionDataSetName

So probably shouldn't set that unless FW >= 5.4.2 .

EDIT: Followed by...

140907:023853 cvm[1922]: E rc:lwsf:http_code=0,handler=process_/change,luaerr=LUA_ERRRUN(2):worker script failed: /usr/lib/ccat/change.lua:1371: /usr/lib/ccat/change.lua:572: Attempt to set unknown field collectionCount

If you say so, mr firmware...

----

Hacking those fields out works, I successfully reimported my moderately heavy db (~500 items), in 46.21s .

EDIT: Woopsie. Except the framework doesn't see anything inside the collections (although the item count is accurate). As for CM, it sees them, but refuses to open them (again, item count is accurate).

The journal shows that everything was imported with a null cdeKey & cdeType...

EDIT²: Ah, gotcha. They were imported without splitting the legacy hash into two fields. Looking into it .

[NiLuJe]

Quick'n dirty 4AM patch that should handle everything...

Is probably terribly ugly.

Should take care of detecting the FW version properly, although someone on FW 5.4.0 should nerf is_cc_aware to check if the patch-level to check for is really 5.4.2 and not 5.4...

That function should probably have a better name, too (is_cloud_aware?).

Left my mess of printfs in.

Also left a bunch of WIP FIXMEs that probably turned out to be stupid and useless ideas on my part. (@barsanuphe: Getting a db uuid from our cdeKey is enough, since the rest of the fields are okay in the db, we only needed to match the cdeKey from the json to a db uuid).
That also means the reasoning behind the last edit in my previous post is somewhat bogus. The 'wrong' part whas ID'ed properly, but my explanation of the fix is wrong (I was thinking in terms of the raw SQL db, and not what ccat does/expects).

TL;DR: Should be better, but in the name of all that is holy, BACK YOUR CC.DB UP FIRST!

----

Did NOT touch the export at all, so it's still in the LibrarianSync format.

----

As for the speed:

Pro: LibrarianSync is super-fast!
Con: The Kindle's JSON over HTTP to SQLite through lua backend isn't. The vast majority of the time the script takes to run is spent waiting for ccat to do its thing.

With ~500 items stored (json db is 45KB), it takes ~40s to complete on a Touch. That's still faster than CM, if memory serves me.

That means we probably should wrap the KUAL button in a shell wrapper with visual feedback of the progress.

----

Anyway, that's it for tonight, gonna treat me to the latest Doctor Who as a reward, because it's already 4AM, so eff it, why the hell not ;p.

[knc1]

OK - the fog has slowly lifted here, since I last posted (and mis-descripted) my own contribution.

The release thread:
https://www.mobileread.com/forums/sho...d.php?t=240616
(you only need the base-* archive for the HF runtime)

The How-To Install it and use it:
https://www.mobileread.com/forums/sho...1&postcount=47
(Either over ssh or from inside kterm)
Note: it installs in */esys rather than under */extensions or */system (a pathlength problem in editing elf files).

The 5cent tour:

core2quad mszick # telnet ken1
Trying 192.168.15.244...
Connected to ken1.morethan.org.
Escape character is '^]'.

Welcome to Kindle!

#################################################
#  N O T I C E  *  N O T I C E  *  N O T I C E  # 
#################################################
Rootfs is mounted read-only. Invoke mntroot rw to
switch back to a writable rootfs.
#################################################
[root@kindle root]# cd /mnt/us
[root@kindle us]# exec esys/bin/busybox ash
[root@kindle us]# export PATH=/mnt/us/esys/bin:/mnt/us/esys/usr/bin:$PATH
[root@kindle us]# lua
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
> os.exit(0)
[root@kindle us]# luajit
LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/
JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse
> os.exit(0)

Although the prompt did not change, the 'exec' replaced your command line interpreter with ash-1.22.1

The first one (Lua) is Amazon's release from their system tree.
The second one (LuaJit) is from the HF system tree.
(Yes Virginia, you can run two (or more) different system library trees on Linux, concurrently (no chroot required) )

Now try running Amazon's Lua extension module(s):

[root@kindle us]# luajit
LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/
JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse
> require 'json'
stdin:1: module 'json' not found:
        no field package.preload['json']
        no file './json.lua'
        no file '/usr/share/luajit-2.0.3/json.lua'
        no file '/usr/local/share/lua/5.1/json.lua'
        no file '/usr/local/share/lua/5.1/json/init.lua'
        no file '/usr/share/lua/5.1/json.lua'
        no file '/usr/share/lua/5.1/json/init.lua'
        no file './json.so'
        no file '/usr/local/lib/lua/5.1/json.so'
        no file '/usr/lib/lua/5.1/json.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
        [C]: in function 'require'
        stdin:1: in main chunk
        [C]: at 0x00009c48
>

Phooey!
Amazon installed that extension library in:

/usr/lib/lua/json.lua

And the default search path is looking in (the proper place):

/usr/share/lua/5.1/json.lua

Note: /usr/lib/*, /usr/local/lib/* is where the binary modules (should) live; json.lua is a source code module.

The same problem will exist for all the Amazon provided library modules.

A quick and very dirty work-around is to copy the module file(s) to the current working directory:

[root@kindle us]# cd
[root@kindle root]# cp -a /usr/lib/lua/json.lua .
[root@kindle root]# luajit
LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/
JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse
> require 'json'
>

But that is tiring and error prone.

I should work up a new 'init' file for LuaJit to make its search paths match the Amazon install.
(A WIP)

Well, the source code paths can be handled by:

[root@kindle root]# luajit
LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/
JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse
> package.path = package.path .. ";/usr/lib/lua/?.lua"
> require 'sqlite3'
> require 'json'
>

And similar for the C binary search paths.

Why?
Because once fixed, this should deal with the "lua ccat is too slow" comments.

Note:
For the interested, the entry point of the ccat extension module is:
/usr/lib/ccat/init.lua

Ref:

[root@kindle root]# busybox
BusyBox v1.22.1 (2014-06-14 09:41:57 CDT) multi-call binary.
[root@kindle root]# cat /etc/prettyversion.txt
Kindle 5.3.3 (~~otaVersion~~)
[root@kindle root]# uname -a
Linux kindle 2.6.31-rt11-lab126 #1 Tue Jan 8 22:06:21 PST 2013 armv7l GNU/Linux
[root@kindle root]# /mnt/us/esys/lib/libc.so.6
GNU C Library (Buildroot) stable release version 2.19, by Roland McGrath et al.
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.9.0.
Compiled on a Linux 3.0.101 system on 2014-06-14.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

[NiLuJe]

@knc1: Handling the SQL ourselves is also a solution to the ccat is slow problem .

@barsanuphe: I'm cleaning up the patch in a GH fork, I'll send a PR when I'm done .

[barsanuphe]

@NiLuJe: I just rewrote most of LibrarianSync (pushed on github), because the code was becoming a little too horrible for my taste. Hopefully it's a little clearer now, or at least it should be easier to modify in the future. I haven't merged in the cdekey stuff : what exactly should go in that field and why?

[barsanuphe]

OK, last commit has support for exporting a json file in the calibre kindle plugin format.
For now, it's created as "exported_collections_calibre_plugin.json" in the extensions folder. Would it be better to overwrite /mnt/us/systems/collections.json?

As for the export in the LibrarianSync format, it could be useful for exporting current collections, in order to modify the json by hand (since it's just labels and paths), then sync back the results.

Anyway, "export" now writes both files at once.

[knc1]

Originally Posted by NiLuJe

@knc1: Handling the SQL ourselves is also a solution to the ccat is slow problem .

I have been editing my post for the last hour.
I think I am done changing it now.

Note: I end with loading sqlite3 and json modules (forgetting entirely about ccat )

[NiLuJe]

@barsanuphe: Argh. .

cf. https://github.com/NiLuJe/librarian for my somewhat cleaned up changes before the refactor.

And, yep, the export should overwrite system/collections.json .

EDIT: As for the cdeKey stuff, the idea is to parse the legacy hash found in the json db to get the proper cdeKey, and then match that cdeKey to the book's uuid in the db, since this is this id that needs to go in a collection, not the cdeKey or the legacy hash.

[barsanuphe]

Originally Posted by NiLuJe

@barsanuphe: Argh. .

cf. https://github.com/NiLuJe/librarian for my somewhat cleaned up changes before the refactor.

And, yep, the export should overwrite system/collections.json .

sorry
Thanks for the additions, I think I'll try to manually merge this into the newly refactored code.
And ok for overwriting the json file.

[NiLuJe]

@barsanuphe: ^^ ;p.

While you're at it, I realized halfway through the shell wrapper that it was probably overkill, if we just wrap the gist of main in a try/except block, we should be able to handle the "success"/"failure" feedback from inside the Python script, and simply get rid of the then unnecessary wrapper.

We'd just loose the idiot-proof python check, but I guess you deserve to get silent failure if you forget to install Python before running a Python script ;p.

[barsanuphe]

ok, I just pushed the merged code + export now overwrites the calibre plugin json. It should work as intended, but the calibre kindle plugin integration wasn't tested with an independent json file (I did an export then import).

About the wrapper: I was just wondering why it was needed instead of try/except, and thought it was about the kh_msg part for some reason (well, mainly because I didn't look at that part)

[NiLuJe]

Yep, that was me running in autopilot before remembering that I already had ported the logging stuff to Python ;p.

I'll check it out later, thanks .