LibrarianSync: Kindle collections builder

[NiLuJe]

Was that time spent building the json db (Updating collections ...) or waiting for ccat (Sending commands ...)? If it's the latter, building the json db from a powerful host won't change much (i.e., ccat is the bottleneck, and since you'll always have to go through it…).

[mobusr]

Originally Posted by NiLuJe

Was that time spent building the json db (Updating collections ...) or waiting for ccat (Sending commands ...)? If it's the latter, building the json db from a powerful host won't change much (i.e., ccat is the bottleneck, and since you'll always have to go through it…).

i have no idea, i just ran KUAL and selected librarysync -> create collection from folders structure and it was displaying something like 'creating collections from directory structure ... ' for 3+ hours..

[barsanuphe]

Then it should be while building the json, going through the folders. It should switch to something like "Sending commands" when doing the actual updating...

[mobusr]

i've also tried - https://github.com/foxsen/collectionSync it seems doesn't work anymore..

@baranuphe, i'm scraping now whole lot of books, it should create a .json i guess?

it seems librarian doesn't like '%' in filenames, can you fix this?

also, is there any way to continue import (not start from scratch) with librarian?

if i run librarian --scrape -i it starts again from the start ... ?

ehm, what's the point of converting books from .mobi to .epub?
when you send to kindle they are converted again back to .mobi?
i kinda don't get for what this is made for ..

would be just nice to get a tool to make same 'generate .json file based on the directory tree' on the desktop PC.

EDIT: i'm just looking into librariansync/generate_collections.py file and trying to adopt it to the PC to generate db on it but i'm not that good in python so help is highly appreciated.

EDIT2: so i've installed usbnet on my PW2 and gave a try CollectionSync script from the telnet it seems to be working but have some bugs (with dashes in name of the folders/files), most of the time it takes to update the final collections on the homescreen of the kindle with some java app called (PerformPostBatch.class) but it works MUCH faster comparing to librarianSync, so roughly 3920 books with 360 collections (actually about 400 folders, but some didnt work of the problem above) took about ~10 minutes.

the other bug I encountered kindle's home crashed as well for some reason, so had to reboot from console )

also it seems to not create sub-collections like CM, e.g. folder1\folder2\some_book.mobi so you get folder1 collection inside it folder2 collection hidden on the home page, but I think its doable to modify the script to make it work properly or just integrate this script functionality into the librarian (which would be quite nice).

[barsanuphe]

So:
1. scraping isn't importing, it'll just copy files scattered in a drive into the import folder.
2. I'll see about the "%" character in filenames
3. importing can be resumed, not scraping.
4. at some point I decided to only have epubs in the library. It's easier to work with as a format and, I guess, by its open nature, more future-proof.
5. you could look at list_folder_contents() in kindle_contents.py to have an idea of how to build your offline script to generate the json.
6. I don't really get why you would get such a difference in terms of speed. Maybe it's a memory thing? how are your files organized? lots of folders/subfolders or a few folders with a lot of files?
7. nested folders do not work in later firmwares, so I dropped support for that during a rewrite.

[mobusr]

Originally Posted by barsanuphe

So:
1. scraping isn't importing, it'll just copy files scattered in a drive into the import folder.
2. I'll see about the "%" character in filenames
3. importing can be resumed, not scraping.
4. at some point I decided to only have epubs in the library. It's easier to work with as a format and, I guess, by its open nature, more future-proof.
5. you could look at list_folder_contents() in kindle_contents.py to have an idea of how to build your offline script to generate the json.
6. I don't really get why you would get such a difference in terms of speed. Maybe it's a memory thing? how are your files organized? lots of folders/subfolders or a few folders with a lot of files?
7. nested folders do not work in later firmwares, so I dropped support for that during a rewrite.

6. Well about 3920 files in 401 folders. I've also made in the root 6 main folders (categories). So structure is documentsROOT -> 6 folders -> 401 folders total. In every subfolder there are files in between of 1 to 200 or so.
I think the main issue is converting mobi to epub.
7. Ahh, didn't know that. What they've changed there? I've looked into /var/local/cc.db and collections structure is pretty much in Collections table, each Collection is in Entries with Collection type. I'm interested in just updating the sqlite3 db and restarting amazon's launcher, is it possible ? (without whole device restart).

[knc1]

Originally Posted by mobusr

- - - -
EDIT2: so i've installed usbnet on my PW2 and gave a try CollectionSync script from the telnet it seems to be working but have some bugs (with dashes in name of the folders/files), most of the time it takes to update the final collections on the homescreen of the kindle with some java app called (PerformPostBatch.class) but it works MUCH faster comparing to librarianSync, so roughly 3920 books with 360 collections (actually about 400 folders, but some didnt work of the problem above) took about ~10 minutes.

- - - -

That is an (intentional) artifact of CollectionSync being written for Chinese.
Details in the "readme" document and change log at the link you posted earlier.

From what I could see at a brief look-through, its not the '-' but the preceding and trailing space in ' - '.
*nix doesn't like spaces in file and path names - you have to escape them or quote the string that has them.

[barsanuphe]

Originally Posted by mobusr

6. Well about 3920 files in 401 folders. I've also made in the root 6 main folders (categories). So structure is documentsROOT -> 6 folders -> 401 folders total. In every subfolder there are files in between of 1 to 200 or so.
I think the main issue is converting mobi to epub.

LibrarianSync, on the Kindle, does not convert anything. It just crawls your folders to create the collections.

[kaznelson]

Originally Posted by barsanuphe

2. I'll see about the "%" character in filenames

Can make automatic replacement of unrecognised symbols in filename on "_"?

PS: I don't have problems from "%"...

[barsanuphe]

@mobusr
I've added a new script to the git repo: tools/offline_folder_export.py that basically creates the collections.json file necessary to build folder-based collections. You can run this on your PC on your local mirror of files, copy your files over to your kindle and collections.json to the extensions folder, and rebuild/update your collections with LibrarianSync.

[mobusr]

Originally Posted by barsanuphe

@mobusr
I've added a new script to the git repo: tools/offline_folder_export.py that basically creates the collections.json file necessary to build folder-based collections. You can run this on your PC on your local mirror of files, copy your files over to your kindle and collections.json to the extensions folder, and rebuild/update your collections with LibrarianSync.

wow thanks! will test this and report back.

[barsanuphe]

@mobusr
What exactly was wrong with having "%" in filenames?

[barsanuphe]

Following a discussion with frakman1, latest version allows for regular expressions in collections.json, for quick sorting into collections of files whose filenames follow known patterns.

[mobusr]

@barsanuphe

I think all my problems related to LibrarianSync is because I synced git repo from Windows system so git added nasty ^M at the end of each line ..

I ran today few hours ago 'Export Collections function' and it ran for over 10 hours.

So I thought it was something wrong and decided to run same command through the ssh..

I had in the terminal something like:

command not found

after editing generate_collections.py to make sure it's /usr/bin/env python instead of /usr/bin/env python2.7 it worked .. and was done after 38.52s...

I'm gonna test your script now for creating collections.json file on localhost.. and report back.. sorry for longness!

[NiLuJe]

@mobusr: Err, that doesn't make sense. When did you install Python, and which version of the package did you use?

EDIT: That is, unless the shebang itself was mangled by carriage returns. But then just dos2unix'ing it should be enough, the python2.7 symlink has been there as long as the python one, so moving from one to the other wouldn't change anything, which is what I was getting at .