![]() |
#31 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
Okay, I moved the relevant posts from the old thread, and merged some stuff to somewhat reorder it properly.
Give me a shout if I broke something ![]() |
![]() |
![]() |
![]() |
#32 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
thanks NiLuJe, everything looks good.
BTW the new export option for LibrarianSync uses my own json format, and not the format used by the calibre kindle plugin. Would that be useful or even make sense? |
![]() |
![]() |
Advert | |
|
![]() |
#33 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@barsanuphe: Both CM and the Calibre Plugin only understand the legacy json schema, so, yeah, unless you also think of something to do with an export inside LibrarianSync itself, yeah it'd probably be more useful in the other format
![]() (Granted, the interaction with CM might not be terribly useful, but the Calibre Plugin features a GUI to manually manage the collections, which some people find useful). I haven't yet checked how it behaves, but, FWIW, the exact filename CM & the Plugin expect is [/mnt/us/]system/collections.json ![]() Last edited by NiLuJe; 09-06-2014 at 08:24 PM. |
![]() |
![]() |
![]() |
#34 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
Okay, testing on a Touch, kablooey!
Code:
140907:023456 cvm[1922]: E rc:lwsf:http_code=0,handler=process_/change,luaerr=LUA_ERRRUN(2):worker script failed: /usr/lib/ccat/change.lua:1371: /usr/lib/ccat/change.lua:1055: Attempt to set unknown field collectionDataSetName ![]() EDIT: Followed by... Code:
140907:023853 cvm[1922]: E rc:lwsf:http_code=0,handler=process_/change,luaerr=LUA_ERRRUN(2):worker script failed: /usr/lib/ccat/change.lua:1371: /usr/lib/ccat/change.lua:572: Attempt to set unknown field collectionCount ---- Hacking those fields out works, I successfully reimported my moderately heavy db (~500 items), in 46.21s ![]() EDIT: Woopsie. Except the framework doesn't see anything inside the collections (although the item count is accurate). As for CM, it sees them, but refuses to open them (again, item count is accurate). The journal shows that everything was imported with a null cdeKey & cdeType... EDIT²: Ah, gotcha. They were imported without splitting the legacy hash into two fields. Looking into it ![]() Last edited by NiLuJe; 09-07-2014 at 01:12 AM. |
![]() |
![]() |
![]() |
#35 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
Quick'n dirty 4AM patch that should handle everything...
Is probably terribly ugly. Should take care of detecting the FW version properly, although someone on FW 5.4.0 should nerf is_cc_aware to check if the patch-level to check for is really 5.4.2 and not 5.4... That function should probably have a better name, too (is_cloud_aware?). Left my mess of printfs in. Also left a bunch of WIP FIXMEs that probably turned out to be stupid and useless ideas on my part. (@barsanuphe: Getting a db uuid from our cdeKey is enough, since the rest of the fields are okay in the db, we only needed to match the cdeKey from the json to a db uuid). That also means the reasoning behind the last edit in my previous post is somewhat bogus. The 'wrong' part whas ID'ed properly, but my explanation of the fix is wrong (I was thinking in terms of the raw SQL db, and not what ccat does/expects). TL;DR: Should be better, but in the name of all that is holy, BACK YOUR CC.DB UP FIRST! ---- Did NOT touch the export at all, so it's still in the LibrarianSync format. ---- As for the speed: Pro: LibrarianSync is super-fast! Con: The Kindle's JSON over HTTP to SQLite through lua backend isn't. The vast majority of the time the script takes to run is spent waiting for ccat to do its thing. With ~500 items stored (json db is 45KB), it takes ~40s to complete on a Touch. That's still faster than CM, if memory serves me. That means we probably should wrap the KUAL button in a shell wrapper with visual feedback of the progress. ---- Anyway, that's it for tonight, gonna treat me to the latest Doctor Who as a reward, because it's already 4AM, so eff it, why the hell not ;p. Last edited by NiLuJe; 09-07-2014 at 06:07 PM. Reason: Remove deprecated attachment ;) |
![]() |
![]() |
Advert | |
|
![]() |
#36 |
Going Viral
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 17,212
Karma: 18210809
Join Date: Feb 2012
Location: Central Texas
Device: No K1, PW2, KV, KOA
|
LuaJit-2.0.3
OK - the fog has slowly lifted here, since I last posted (and mis-descripted) my own contribution.
![]() The release thread: https://www.mobileread.com/forums/sho...d.php?t=240616 (you only need the base-* archive for the HF runtime) The How-To Install it and use it: https://www.mobileread.com/forums/sho...1&postcount=47 (Either over ssh or from inside kterm) Note: it installs in */esys rather than under */extensions or */system (a pathlength problem in editing elf files). The 5cent tour: Code:
core2quad mszick # telnet ken1 Trying 192.168.15.244... Connected to ken1.morethan.org. Escape character is '^]'. Welcome to Kindle! ################################################# # N O T I C E * N O T I C E * N O T I C E # ################################################# Rootfs is mounted read-only. Invoke mntroot rw to switch back to a writable rootfs. ################################################# [root@kindle root]# cd /mnt/us [root@kindle us]# exec esys/bin/busybox ash [root@kindle us]# export PATH=/mnt/us/esys/bin:/mnt/us/esys/usr/bin:$PATH [root@kindle us]# lua Lua 5.1.4 Copyright (C) 1994-2008 Lua.org, PUC-Rio > os.exit(0) [root@kindle us]# luajit LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/ JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse > os.exit(0) The first one (Lua) is Amazon's release from their system tree. The second one (LuaJit) is from the HF system tree. (Yes Virginia, you can run two (or more) different system library trees on Linux, concurrently (no chroot required) ![]() Now try running Amazon's Lua extension module(s): Code:
[root@kindle us]# luajit LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/ JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse > require 'json' stdin:1: module 'json' not found: no field package.preload['json'] no file './json.lua' no file '/usr/share/luajit-2.0.3/json.lua' no file '/usr/local/share/lua/5.1/json.lua' no file '/usr/local/share/lua/5.1/json/init.lua' no file '/usr/share/lua/5.1/json.lua' no file '/usr/share/lua/5.1/json/init.lua' no file './json.so' no file '/usr/local/lib/lua/5.1/json.so' no file '/usr/lib/lua/5.1/json.so' no file '/usr/local/lib/lua/5.1/loadall.so' stack traceback: [C]: in function 'require' stdin:1: in main chunk [C]: at 0x00009c48 > Amazon installed that extension library in: Code:
/usr/lib/lua/json.lua Code:
/usr/share/lua/5.1/json.lua The same problem will exist for all the Amazon provided library modules. A quick and very dirty work-around is to copy the module file(s) to the current working directory: Code:
[root@kindle us]# cd [root@kindle root]# cp -a /usr/lib/lua/json.lua . [root@kindle root]# luajit LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/ JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse > require 'json' > I should work up a new 'init' file for LuaJit to make its search paths match the Amazon install. (A WIP) Well, the source code paths can be handled by: Code:
[root@kindle root]# luajit LuaJIT 2.0.3 -- Copyright (C) 2005-2014 Mike Pall. http://luajit.org/ JIT: ON ARMv7 VFPv3 fold cse dce fwd dse narrow loop abc sink fuse > package.path = package.path .. ";/usr/lib/lua/?.lua" > require 'sqlite3' > require 'json' > Why? Because once fixed, this should deal with the "lua ccat is too slow" comments. Note: For the interested, the entry point of the ccat extension module is: /usr/lib/ccat/init.lua Ref: Code:
[root@kindle root]# busybox BusyBox v1.22.1 (2014-06-14 09:41:57 CDT) multi-call binary. [root@kindle root]# cat /etc/prettyversion.txt Kindle 5.3.3 (~~otaVersion~~) [root@kindle root]# uname -a Linux kindle 2.6.31-rt11-lab126 #1 Tue Jan 8 22:06:21 PST 2013 armv7l GNU/Linux [root@kindle root]# /mnt/us/esys/lib/libc.so.6 GNU C Library (Buildroot) stable release version 2.19, by Roland McGrath et al. Copyright (C) 2014 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Compiled by GNU CC version 4.9.0. Compiled on a Linux 3.0.101 system on 2014-06-14. Available extensions: crypt add-on version 2.1 by Michael Glad and others GNU Libidn by Simon Josefsson Native POSIX Threads Library by Ulrich Drepper et al BIND-8.2.3-T5B libc ABIs: UNIQUE For bug reporting instructions, please see: <http://www.gnu.org/software/libc/bugs.html>. Last edited by knc1; 09-07-2014 at 11:44 AM. |
![]() |
![]() |
![]() |
#37 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@knc1: Handling the SQL ourselves is also a solution to the ccat is slow problem
![]() @barsanuphe: I'm cleaning up the patch in a GH fork, I'll send a PR when I'm done ![]() |
![]() |
![]() |
![]() |
#38 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
@NiLuJe: I just rewrote most of LibrarianSync (pushed on github), because the code was becoming a little too horrible for my taste. Hopefully it's a little clearer now, or at least it should be easier to modify in the future. I haven't merged in the cdekey stuff : what exactly should go in that field and why?
|
![]() |
![]() |
![]() |
#39 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
OK, last commit has support for exporting a json file in the calibre kindle plugin format.
For now, it's created as "exported_collections_calibre_plugin.json" in the extensions folder. Would it be better to overwrite /mnt/us/systems/collections.json? As for the export in the LibrarianSync format, it could be useful for exporting current collections, in order to modify the json by hand (since it's just labels and paths), then sync back the results. Anyway, "export" now writes both files at once. |
![]() |
![]() |
![]() |
#40 |
Going Viral
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 17,212
Karma: 18210809
Join Date: Feb 2012
Location: Central Texas
Device: No K1, PW2, KV, KOA
|
|
![]() |
![]() |
![]() |
#41 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@barsanuphe: Argh.
![]() cf. https://github.com/NiLuJe/librarian for my somewhat cleaned up changes before the refactor. And, yep, the export should overwrite system/collections.json ![]() EDIT: As for the cdeKey stuff, the idea is to parse the legacy hash found in the json db to get the proper cdeKey, and then match that cdeKey to the book's uuid in the db, since this is this id that needs to go in a collection, not the cdeKey or the legacy hash. Last edited by NiLuJe; 09-07-2014 at 12:05 PM. |
![]() |
![]() |
![]() |
#42 | |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
Quote:
![]() Thanks for the additions, I think I'll try to manually merge this into the newly refactored code. And ok for overwriting the json file. |
|
![]() |
![]() |
![]() |
#43 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
@barsanuphe: ^^ ;p.
While you're at it, I realized halfway through the shell wrapper that it was probably overkill, if we just wrap the gist of main in a try/except block, we should be able to handle the "success"/"failure" feedback from inside the Python script, and simply get rid of the then unnecessary wrapper. We'd just loose the idiot-proof python check, but I guess you deserve to get silent failure if you forget to install Python before running a Python script ;p. |
![]() |
![]() |
![]() |
#44 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 77
Karma: 40008
Join Date: Aug 2014
Location: Paris, France
Device: PW2, Kobo H20, Kobo Aura One
|
ok, I just pushed the merged code + export now overwrites the calibre plugin json. It should work as intended, but the calibre kindle plugin integration wasn't tested with an independent json file (I did an export then import).
About the wrapper: I was just wondering why it was needed instead of try/except, and thought it was about the kh_msg part for some reason (well, mainly because I didn't look at that part) ![]() |
![]() |
![]() |
![]() |
#45 |
BLAM!
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 13,492
Karma: 26047188
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E
|
Yep, that was me running in autopilot before remembering that I already had ported the logging stuff to Python ;p.
I'll check it out later, thanks ![]() |
![]() |
![]() |
![]() |
Tags |
kindle collections, librariansync |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Vocabulary builder and flash card feature on old kindle 2/3/dx? | dark.knight1 | Kindle Developer's Corner | 12 | 10-10-2015 01:12 PM |
Vocabulary builder | bobafruit | Kindle Developer's Corner | 25 | 09-04-2015 01:46 AM |
Free (nook/Kindle/iTunes/DRM-free) Legacy Builder [Xtian Business Leadership Advice] | ATDrake | Deals and Resources (No Self-Promotion or Affiliate Links) | 0 | 10-13-2014 02:44 AM |
Vocabulary Builder for Kindle Paperwhite 1 | pxpxpx | Amazon Kindle | 2 | 02-16-2014 06:07 PM |
SBPubX Builder | tmaynard | Fictionwise eBookwise | 16 | 11-20-2008 12:35 PM |