LibrarianSync: Kindle collections builder

[stefano.sb]

Originally Posted by NiLuJe

Nope, might have more time in spring, though.

Hi…

Stefano

[midrare]

I want to take a look at updating this project for Paperwhite SE. Could you give a rough outline of what I need to do? What still needs investigating and what parts need updating?

[NiLuJe]

The code changes are probably irrelevant (and minor, probably just the device IDs), it's quite likely blocked by the lack of an updated FBInk/Python build with support for the new devices.

[wow400]

Originally Posted by NiLuJe

I haven't had time to rebuild stuff for the PW5 yet .

Hi NiLuJe, any news on the PW5 update please?

[NiLuJe]

I'm hoping to get in done before fall.

[Georgie W]

I've got two books that will not stay in their collection & I'm baffled.

Background - Books 1 & 2 are by XYZ. I've got a collection that I've allocated them to in Calibre, called XYZ. They are the only two books in this collection, and when the problem started I hadn't read or opened either of them for a long time. When I run Kindle Collections in Calibre, the log report shows them in their proper collection.

Problem - When I go to my Kindle - Open KUAL - Librarian Sync - Update folders - any new books I've added are sorted into their proper places, but Books 1 & 2 aren't - they are "Floating" at the end of the list of collections in collections view. Collection XYZ is still there but now has no books in it. I can manually add 1&2 to it, and they'll stay there, but only till the next time I run Update folders; after that, they are back to floating again.

I've tried
1. deleting the collection in the Kindle & then recreating it. That works for the first time, then the books slip out of the collection again when I next Update;
2. reconverting the books, I've tried taking them out of Calibre entirely then re-adding as if they were new;
3. running disc checkers on the Kindle to check I've not got a bad sector or whatever.

This has occasionally happened in the past to a random book or so, but never with the persistence that 1&2 are showing!

Any thoughts? Thanks.

[wow400]

Originally Posted by NiLuJe

I'm hoping to get in done before fall.

Thank you very much

[knolle75]

I got following message in mrinstaller.log on a PW5 (2021), device code 21A, FW 5.16.2.1.1.

Spoiler:

**** **** **** ****

[2023-11-09 @ 22:05:44 +0058] :: [MRPI r19303] - Beginning the processing of package 'Update_librariansync_20201109_kindle5.bin' (librariansync 20201109 kindle5) . . .


Package 'Update_librariansync_20201109_kindle5.bin' (librariansync 20201109 kindle5) is not targeting your device [82A vs. 11
0F
10
24
1B
1C
1D
1F
20
D4
5A
D5
D6
D7
D8
F2
17
60
62
5F
61
C6
DD
13
54
53
2A
52
201
207
206
202
205
204
293
26B
26C
294
26E
26F
20C
20D
21C
219
21B
269
26A
2E8
341
34A
342
2E1
2E6
2F7
366
361
362
402
363
4DB
4D8
4DC
4DD
403
4D9
4DA
414
3D1
3AB
3CF
434
3D8
3D6
3D5
3D4
3D7], skipping . . .

I hope that can be easily fixed. Thanks in advance

Update: After checking out the first options in KOreader, which I started to be in love with, I understood how books are organized there. Simply folder wise, which is exactly what my wife loves. On top you can use the meta data library of Calibre in KOreader to find books in another way.
Last but not least, using my Raspberry as Calibre server and connect it with OPDS is the cherry on the cake.

Thanks to all who spend/spent there personal time to make user like me happy. I will do some donations in the coming days to honor that.

[stefano.sb]

Originally Posted by wow400

Hi NiLuJe, any news on the PW5 update please?

@NiLuJe this update is the only reason why I have jailbreak my pw5

Stefano

[wow400]

Originally Posted by NiLuJe

I'm hoping to get in done before fall.

Any news at all please?

[bigwoof]

Originally Posted by wow400

Any news at all please?

Hi,

I just got librariansync working on a brand new PW5 (paperwhite 2021 signature edition bought during this 2023 BF. it's so new a new device ID needed to be added for it in kindletool. Thanks NiLuJe!!)

;tldr it all just works with one missing autodetect feature and a key "install python 2.7 pacakge on kindle" support step (step 3 below) needed.

basically do the following.

1. on the Calibre side, install Kindle Collections, do the QT6 patch if you plan to use the manual edit function (otherwise ignore it. look at the last few pages of the extension page for the patch)

2. the autodetect feature of Kindle Collections will fail when you do the "Setup" step. just select something. I used the paperwhite setting and it works

3. on the kindle side, install the librarian sync (using mrpi) *AND* the python 2.7 PW5 package from the snapshots page. If you don't install python, librariansync will just keep failing *silently* over and over again.

4. done. if you have synced the collections in calibre and got the "run librariansync update on your kindle" popup message from the calibre plugin, just run that menu option on the kindle, wait a few seconds (there will be no "finised message"), tap the go back option at the bottom (the /), go to your library screen and watch all the collections show up..

original "debugging" thread and post is here btw

https://www.mobileread.com/forums/sh...92#post4381692

[bigwoof]

found another bug that needs fixing. The sqlite DB barfs when too many books are added at one go. the error is

 { "Error": "Failed to execute handler: /usr/lib/ccat/change.lua:2201: /usr/lib/ccat/change.lua:2045: {"http_status_code":400,"sqlite3_msg":"too many SQL variables","sqlite3_code":4}" }

The code is in cc_update.py

            r = requests.post("http://127.0.0.1:9101/change",
                              data=json.dumps(full_command),
                              headers={'AuthToken': self.session_token, 'content-type
                              proxies={'no': 'pass'})

I'll need to break the data up into groups of maybe 400 and submit that bit by bit

I'm not great at JSON and even worse at SQL so if someone can crank this up in 5 minutes (some folks I know could!), please go ahead. otherwise I'll fix this as a weekend project sometime.. :-)

[bigwoof]

had some time and this was more interesting than doing other things. So I took a look at the code..

It might be as simple as doing this in generate_collections.py

       for i in range(0,len(collection.ebooks),400):
             new_collection = collection.ebooks[i:i+400]
             cc.update_collections_entry(collection.uuid,
                                         [e.uuid for e in new_collection])

that seems to break up the long members array into groups of 400 with all the correct preambles preserved.. will test it tomorrow. Hope that I don't need to send less commands at one go. assuming that the issue is the length of a single command and not the total number of small commands sent.. will find out tomorrow in any case..

[bigwoof]

first tests are promising. adding a 2000 book single collection now works where it used to fail previously.

going to try with 4000+ books and 100+ collections now.

Update:

and it did not work. The kindle rebooted after printing

   231221:074522 system: I LibrarianSync:cc_update:: Sending commands...
tail: /var/log/messages has been replaced; following end of new file
   231221:074645 system: E LibrarianSync:main:: Something went very wrong.

might be too many SQL commands to process at a go. I'll try sending 10 SQL commands at a time and see if that works better..

UPDATE: doing the SQL commands in batches of 100 and it seems to be working so far.

and it succeeded.. but the collections in the kindle don't have the right number of entries.. will need to dig into the SQL being sent and see where I messed up.. probably need to change the "INSERT" into "UPDATE" for the items that were broken into multiple SQL commands..

the code I am using is here btw. if anyone wants to try it.

made changes at line 117 in cc_update.py (to split up the number of SQL commands sent in each POST message)

and at line 237 in generate_collections.py (to break up the items in each SQL command). added debugging output to see the books being added as well.

[bigwoof]

found the bug. The error is in this function in cc_update.py

    def update_collections_entry(self, coll_uuid, members):
        self.commands.append(
            {
                "update":
                    {
                        "type": "Collection",
                        "uuid": str(coll_uuid),
                        "members": members
                    }
            })

I need to change the update into an append now that I am breaking up the argument list for members. probably using something like this

https://stackoverflow.com/questions/...xt-row-sqlite3