LibrarianSync: Kindle collections builder

[kaznelson]

Originally Posted by knc1

Are those "cloud" collections you are trying to export or "local" collections?

"Local" collections - created by LibrarianSync from folders...

[NiLuJe]

Yup, confirmed.

Trivial fix in PR #3 .

(That was a regression, which is why I remembered it as working fine ;p).

EDIT: Well, that was the shortest-lived PR ever. xD. git master should be working fine now .

[barsanuphe]

Thanks for the report and the PR! The fix is now merged.

Edit: added v1.6 in first post.

[kaznelson]

NiLuJe, barsanuphe
Great! It's worked; collections files are correct.
Thanks!

[mobusr]

thanks for the plugin!

i've tired using CM for building collections based on the folder structure (took so long ~12-16 hours for 3920 books).

hopefully your plugin will be much faster!

what i really wonder is it possible to add an ability to calibre plug-in to build collections based on dir structure on local PC and then sync to kindle to avoid waiting and processing on slow kindle PW device?

the above question is probably more to @NiLuJe.

on another note: is there any tweaks/tips to generally improve performance of plugin on kindle pw2? (e.g. installing gawk will speed it a bit or it doesnt matter at all?)

[NiLuJe]

@mobusr: In this instance (collections from directory structure), the limiting factor is the crappy flash memory used by the PW (i.e., IO), not the CPU, so doing this from a desktop or the device itself shouldn't be drastically different (in LS's case. In CM case there are Java mysteries on top...) [especially since we'd be doing it with code in the exact same language (Python)].

The only potential issue that I can see is that you *might* hit memory consumption issues with very, very large data sets when doing that on the device, but if you have that much entries, the device would already have been slowed down to a crawl anyway ;D.

gawk will speed up KUAL, nothing else.

[knc1]

@mobusr: In addition to NiLuJe's reply directly above - -

The collections catalog is stored in a high level database.

The application using a high level database has very little (usually none) control over how it does its underlying I.O.

The Amazon build of sqlite3 **should** already be as optimized as it can be in regards to I.O. to flash storage.
And if it isn't, we can't do much about that.

Since the initial creation of a large collections catalog is a "one time" operation - -
I think we (the users) will just have to put up with its creation time.

[mobusr]

Originally Posted by NiLuJe

@mobusr: In this instance (collections from directory structure), the limiting factor is the crappy flash memory used by the PW (i.e., IO), not the CPU, so doing this from a desktop or the device itself shouldn't be drastically different (in LS's case. In CM case there are Java mysteries on top...) [especially since we'd be doing it with code in the exact same language (Python)].

The only potential issue that I can see is that you *might* hit memory consumption issues with very, very large data sets when doing that on the device, but if you have that much entries, the device would already have been slowed down to a crawl anyway ;D.

gawk will speed up KUAL, nothing else.

thanks for clarification, but what I meant:

1) sync kindle documents stuff to the PC
2) create same collection db on the PC
3) sync db from the PC to the kindle

Is it possible?

I think this should speed up process significantly.

CM few times shutted down I think because of the memory issues.

[mobusr]

Originally Posted by knc1

@mobusr: In addition to NiLuJe's reply directly above - -

The collections catalog is stored in a high level database.

The application using a high level database has very little (usually none) control over how it does its underlying I.O.

The Amazon build of sqlite3 **should** already be as optimized as it can be in regards to I.O. to flash storage.
And if it isn't, we can't do much about that.

what about making sqlite3 db on the PC and than just syncing it to the kindle?

Since the initial creation of a large collections catalog is a "one time" operation - -
I think we (the users) will just have to put up with its creation time.

Only if there is a mechanizm of updating existing db with new directories structures or updated directories. If you update directories you've got to check whole db to make sure nothing changed / deleted.

[knc1]

Originally Posted by mobusr

what about making sqlite3 db on the PC and than just syncing it to the kindle?

If you truely mean 'sync' then nothing is changed, your relying on the sqlite I.O. limitations.

If you mean copying the database file in its entirety -
That would make it possible to control the low-level I.O. during the copy application.
But that opens a whole new level of complexity -
That database file might be "open" by various applications.

All uses of the "deleted" copy would have to be forced closed before the actual underlying storage was released and the new copy was used.
A system re-start would almost certainly be required.

- - - -

The combination of Librarian and LibrarianSync in combination with Calibre should already handle partial updates.

[NiLuJe]

@mobusr: That's simply not how either CM, LS, nor the Calibre plugin are engineered (and for good reason, as knc1 gave an example of). And I stand by my point that the ends probably wouldn't justify the means . (i.e.: way too much work to achieve a broken-by-design design that wouldn't net any severe speed advantages, but would net us severe downsides).

(Plus, the Kindle DB is *NOT* accessible over USBMS, which makes transparent sync unachievable without a proxy).


As for LS, without doing completely stupid things in the name of speed, the current major hotspot is ccat, and we have no real control over it. Bypassing it (which means reinventing a better wheel than Amazon's) and doing the SQL ourselves would *possibly* net some minor speed advantages, but would increase complexity and decrease maintainability of the code. You're welcome to try your hand at it, LS is modular enough to make that switch doable.


Note that if you use Librarian to manage your library & collections, the first point is moot, because the json will be generated on the library host machine, if I'm not mistaken. The ccat thing still stands, though.
(With a bit of creativity, the same effect can be achieved with the Calibre plugin, since you can use the exact same kind of variables to build the collections as you can to build the directory structure. With the added bonus that both Calibre and the plugin maintain their own cache to avoid IO-bound performance penalties on subsequent runs).


As for CM, there's an extra layer of indirection through the java framework, so who knows how that affects performance. Just try it with LS, it should already be a magnitude faster.


As knc1 ended on, either Librarian + LS, Calibre + Plugin + LS, or LS on its own is already smart enough to do what you want in an efficient manner.

[barsanuphe]

I'm coming a little late to the party, but you could as was mentioned use librarian or at least, in your case, you could write a very small script to list your files on your PC, preserving your folder structure, and generate a json file in the librarian format.
Then you could use LibrarianSync to just perform the collection update without accessing the kindle files.

[mobusr]

thanks guys!

i will try all of the solutions you've posted.

much appreciated!

what if i generate relevant json? how do i import it into kindle's db?

EDIT: re-read @barsnuphe reply, will try that!

[barsanuphe]

By the way, I would be interested to know how long LibrarianSync takes to update your almost 4000 ebooks... from the folder structure or from json.
I have not yet tested librarian & LibrarianSync with that many books.

[mobusr]

Originally Posted by barsanuphe

By the way, I would be interested to know how long LibrarianSync takes to update your almost 4000 ebooks... from the folder structure or from json.
I have not yet tested librarian & LibrarianSync with that many books.

I've started earlier that process on the PW2 and it took 3+ hours and still wasnt finished.

So I'm gonna install librarian into virtual machine with Linux and give it a go.