[GUI Plugin] X-Ray Creator

[stoduk]

Hi szarroug3

As someone mentioned, there is an existing plugin (that mjwilson created, and I've been working on slowly in the background).

I thought I'd give your plugin a run to see how it compares, whether there is any point carrying on with both etc. Not really come to any conclusion on that - the Windows app is far and away the most stable and feature rich, both plugins have promise but have a lot of catching up to do.

[FWIW mine can create decent Xray files for me now, though I've really only tested with new format files. It doesn't yet create the Start/End/Author files, though I was going to add those soon]

Hit a few problems, thought I'd post them here. If you'd rather I submitted as issues on github I can do that. I've tested in two setups:
- OS X 10.10.5 with Calibre 2.56.0
- Windows 7 (under VirtualBox) with Calibre 2.56

Full gory details below. As it stands I haven't been able to get a working Xray using your plugin, but as it works for you I imagine I've got some small difference in my setup.

Cheers,
Anthony

1) Version 2.0.0 in Windows 7
"Selecting 'Book preferences" shows a dialog with only ASIN and shelfari URL (ie. missing the three buttons - search for ASIN, etc.)

2) Version 2.2.0
This is linked from github, but doesn't seem to exist!

2) Version 2.1.0 in Windows 7
a) Crash with book amazon likely doesn't know about
Spotted there was an updated version on the forum, tried this. Problem #1 now gone (fixed or hiding? You'll know more than me), but I get this error when running on Calibre's Quick Start Guide (both text boxes in dialog are empty, I assume because we are barfing trying to find the ASIN)

calibre, version 2.56.0
ERROR: Unhandled exception: <b>AttributeError</b>:'BookConfigWidget' object has no attribute '_asin_edit'

calibre 2.56 embedded-python: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: X-Ray Creator
Traceback (most recent call last):
File "calibre_plugins.xray_creator.book_config", line 173, in search_for_asin
AttributeError: 'BookConfigWidget' object has no attribute '_asin_edit'

b) Lookup problems with A Christmas Carol/lack of error reporting
ASIN B00Q0LB318 is found (think that is coming from book as it is what I'm expecting), but the shelfari URL is 38316941. I thought things were failing here, but a little digging shows that this shelfari entry just has no details at all - so the "aliases" box is being populated with what it finds (ie. nothing!).

Some better status reporting here would go a long way - it isn't clear what happened (ie. shelfari page fetched but nothing found)

c) Fixing the shelfari URL to 4575254 (an entry with decent character data)
I edited the shelfari URL to use a better ID - but this caused Calibre to go in to extreme slow mode. A few seconds for each character press to be recognised. One time your plugin window closed without warning, the second time it worked - and I get a list of Characters to edit!

[Note: I tried to paste the URL in once, instead of typing, and that caused Calibre to lock up completely. Assume this is a similar problem]

Is your plugin doing anything while the "Shelfari URL" text box is being edited? I wonder if edit_shelfari_url() is being run with each character press - if so that could be sllllloooowww. Please don't do that, just update once the click button has been pressed (or after a few moments of inactivity)

d) Making an xray finally
The x-ray that is generated (note: I didn't edit aliases, so we've just got the default character names), and I get garbage. I'm sure Peter is a nice guy, but he really isn't a main character, and certainly he isn't mentioned 65870 times (compared to zero for everyone else!). [I'm sure when I ran this yesterday I got the same result, but with Fred getting all the action]

sqlite> select id, label, count from entity;
id label count
---------- ---------- ----------
0
2 Jacob Marl 0
3 Bob Cratch 0
4 Ghost of C 0
5 The Ghost 0
6 The Ghost 0
7 Mrs. Cratc 0
8 Tiny Tim 0
9 Martha Cra 0
10 Peter Crat 65870
11 Belinda Cr 0
12 Mr. Fezziw 0
13 Mrs. Fezzi 0
14 Ebenezer S 0
15 Dick Wilki 0
16 Fred 0
17 Jacob Marl 0
18 Joe Miller 0
19 Caroline 0
20 Topper 0
21 Belle 0
22 Fan 0
23 Almshouse 0
24 Blind man' 0
25 Humbug 0
sqlite>

e) editing aliases
I tried adding a single alias ("Scrooge" for the main character) and that had no effect - still Peter Cratchit gets all the action, as in previous case.

3) Latest github version in OS X
a) unmodified
I cloned your github and tried that out in OS X - it doesn't work, as I see mentioned in the forum. It would be a shame to not support all OS, especially when the current problem is so minor. You just need to figure out a way to look for external devices in OS X/linux - I suspect something with 'mount' would work.

b) hacked to workaround problem

4) general comments
a) error reporting
you seem to be swallowing some errors, instead of printing the original python stacktrace. eg. I think I got the "windll" assertion with the 2.0.0 version of the plugin, but I get something more cryptic with the github version (KeyError for 'send_status'). That might make debugging harder..

b) interacting with calibre DB
I had been putting this step off, as I hadn't figured out how to do it. I *think* what you are doing is considered naughty, you shouldn't be modifying the Calibre DB under the covers. I meant to start a thread on how to fix that, hopefully Kovid or other has some ideas.

[I think if a book gets renamed or deleted, you'll be leaving junk scattered around the filesystem. Not a terminal problem, but something to be aware of]

c) interacting with kindle
As with (b) I hadn't figured this out. I wondered if Kindle driver would need to be modified. Again, I was going to ask the experts.

[what you are currently doing is duplicating code from Kindle driver, but it should be harmless as you'll only be doing what I do by hand currently]

d) modifying ASIN in books directly
If I understand your code you are modifying the MOBI on the Kindle device directly to update the ASIN. This seems odd! FWIW the "Quality Check" plugin can update ASIN as needed, so might be better to just call that to fix books in the Calibre DB directly.

[szarroug3]

Originally Posted by stoduk

Hi szarroug3

As someone mentioned, there is an existing plugin (that mjwilson created, and I've been working on slowly in the background).

I thought I'd give your plugin a run to see how it compares, whether there is any point carrying on with both etc. Not really come to any conclusion on that - the Windows app is far and away the most stable and feature rich, both plugins have promise but have a lot of catching up to do.

[FWIW mine can create decent Xray files for me now, though I've really only tested with new format files. It doesn't yet create the Start/End/Author files, though I was going to add those soon]

Hit a few problems, thought I'd post them here. If you'd rather I submitted as issues on github I can do that. I've tested in two setups:
- OS X 10.10.5 with Calibre 2.56.0
- Windows 7 (under VirtualBox) with Calibre 2.56

Full gory details below. As it stands I haven't been able to get a working Xray using your plugin, but as it works for you I imagine I've got some small difference in my setup.

Cheers,
Anthony

1) Version 2.0.0 in Windows 7
"Selecting 'Book preferences" shows a dialog with only ASIN and shelfari URL (ie. missing the three buttons - search for ASIN, etc.)

2) Version 2.2.0
This is linked from github, but doesn't seem to exist!

2) Version 2.1.0 in Windows 7
a) Crash with book amazon likely doesn't know about
Spotted there was an updated version on the forum, tried this. Problem #1 now gone (fixed or hiding? You'll know more than me), but I get this error when running on Calibre's Quick Start Guide (both text boxes in dialog are empty, I assume because we are barfing trying to find the ASIN)

calibre, version 2.56.0
ERROR: Unhandled exception: <b>AttributeError</b>:'BookConfigWidget' object has no attribute '_asin_edit'

calibre 2.56 embedded-python: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: X-Ray Creator
Traceback (most recent call last):
File "calibre_plugins.xray_creator.book_config", line 173, in search_for_asin
AttributeError: 'BookConfigWidget' object has no attribute '_asin_edit'

b) Lookup problems with A Christmas Carol/lack of error reporting
ASIN B00Q0LB318 is found (think that is coming from book as it is what I'm expecting), but the shelfari URL is 38316941. I thought things were failing here, but a little digging shows that this shelfari entry just has no details at all - so the "aliases" box is being populated with what it finds (ie. nothing!).

Some better status reporting here would go a long way - it isn't clear what happened (ie. shelfari page fetched but nothing found)

c) Fixing the shelfari URL to 4575254 (an entry with decent character data)
I edited the shelfari URL to use a better ID - but this caused Calibre to go in to extreme slow mode. A few seconds for each character press to be recognised. One time your plugin window closed without warning, the second time it worked - and I get a list of Characters to edit!

[Note: I tried to paste the URL in once, instead of typing, and that caused Calibre to lock up completely. Assume this is a similar problem]

Is your plugin doing anything while the "Shelfari URL" text box is being edited? I wonder if edit_shelfari_url() is being run with each character press - if so that could be sllllloooowww. Please don't do that, just update once the click button has been pressed (or after a few moments of inactivity)

d) Making an xray finally
The x-ray that is generated (note: I didn't edit aliases, so we've just got the default character names), and I get garbage. I'm sure Peter is a nice guy, but he really isn't a main character, and certainly he isn't mentioned 65870 times (compared to zero for everyone else!). [I'm sure when I ran this yesterday I got the same result, but with Fred getting all the action]

sqlite> select id, label, count from entity;
id label count
---------- ---------- ----------
0
2 Jacob Marl 0
3 Bob Cratch 0
4 Ghost of C 0
5 The Ghost 0
6 The Ghost 0
7 Mrs. Cratc 0
8 Tiny Tim 0
9 Martha Cra 0
10 Peter Crat 65870
11 Belinda Cr 0
12 Mr. Fezziw 0
13 Mrs. Fezzi 0
14 Ebenezer S 0
15 Dick Wilki 0
16 Fred 0
17 Jacob Marl 0
18 Joe Miller 0
19 Caroline 0
20 Topper 0
21 Belle 0
22 Fan 0
23 Almshouse 0
24 Blind man' 0
25 Humbug 0
sqlite>

e) editing aliases
I tried adding a single alias ("Scrooge" for the main character) and that had no effect - still Peter Cratchit gets all the action, as in previous case.

3) Latest github version in OS X
a) unmodified
I cloned your github and tried that out in OS X - it doesn't work, as I see mentioned in the forum. It would be a shame to not support all OS, especially when the current problem is so minor. You just need to figure out a way to look for external devices in OS X/linux - I suspect something with 'mount' would work.

b) hacked to workaround problem

4) general comments
a) error reporting
you seem to be swallowing some errors, instead of printing the original python stacktrace. eg. I think I got the "windll" assertion with the 2.0.0 version of the plugin, but I get something more cryptic with the github version (KeyError for 'send_status'). That might make debugging harder..

b) interacting with calibre DB
I had been putting this step off, as I hadn't figured out how to do it. I *think* what you are doing is considered naughty, you shouldn't be modifying the Calibre DB under the covers. I meant to start a thread on how to fix that, hopefully Kovid or other has some ideas.

[I think if a book gets renamed or deleted, you'll be leaving junk scattered around the filesystem. Not a terminal problem, but something to be aware of]

c) interacting with kindle
As with (b) I hadn't figured this out. I wondered if Kindle driver would need to be modified. Again, I was going to ask the experts.

[what you are currently doing is duplicating code from Kindle driver, but it should be harmless as you'll only be doing what I do by hand currently]

d) modifying ASIN in books directly
If I understand your code you are modifying the MOBI on the Kindle device directly to update the ASIN. This seems odd! FWIW the "Quality Check" plugin can update ASIN as needed, so might be better to just call that to fix books in the Calibre DB directly.

Anthony,

First, thanks for the review! It's super helpful.

1) Version 2.0.0 in Windows 7
"Selecting 'Book preferences" shows a dialog with only ASIN and shelfari URL (ie. missing the three buttons - search for ASIN, etc.)

- Version 2.0.0 doesn't have the 3 buttons. I added that in 2.2.0 (which I'll talk about in #2)

2) Version 2.2.0
This is linked from github, but doesn't seem to exist!

- Thanks for catching this. I must've forgotten to push the file to github :/

2) Version 2.1.0 in Windows 7
a) Crash with book amazon likely doesn't know about
Spotted there was an updated version on the forum, tried this. Problem #1 now gone (fixed or hiding? You'll know more than me), but I get this error when running on Calibre's Quick Start Guide (both text boxes in dialog are empty, I assume because we are barfing trying to find the ASIN)

calibre, version 2.56.0
ERROR: Unhandled exception: <b>AttributeError</b>:'BookConfigWidget' object has no attribute '_asin_edit'

calibre 2.56 embedded-python: True is64bit: False
Windows-7-6.1.7601-SP1 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: X-Ray Creator
Traceback (most recent call last):
File "calibre_plugins.xray_creator.book_config", line 173, in search_for_asin
AttributeError: 'BookConfigWidget' object has no attribute '_asin_edit'

- I'll have to look into this. I haven't run across that in 2.1.0 but I'll try it.
Update: turns out I had a typo in the file. Fixed it

b) Lookup problems with A Christmas Carol/lack of error reporting
ASIN B00Q0LB318 is found (think that is coming from book as it is what I'm expecting), but the shelfari URL is 38316941. I thought things were failing here, but a little digging shows that this shelfari entry just has no details at all - so the "aliases" box is being populated with what it finds (ie. nothing!).

Some better status reporting here would go a long way - it isn't clear what happened (ie. shelfari page fetched but nothing found)

- Yes, I've been really bad about status reporting in the book preferences. I'll have to add a status message to these.

c) Fixing the shelfari URL to 4575254 (an entry with decent character data)
I edited the shelfari URL to use a better ID - but this caused Calibre to go in to extreme slow mode. A few seconds for each character press to be recognised. One time your plugin window closed without warning, the second time it worked - and I get a list of Characters to edit!

[Note: I tried to paste the URL in once, instead of typing, and that caused Calibre to lock up completely. Assume this is a similar problem]

Is your plugin doing anything while the "Shelfari URL" text box is being edited? I wonder if edit_shelfari_url() is being run with each character press - if so that could be sllllloooowww. Please don't do that, just update once the click button has been pressed (or after a few moments of inactivity)

- Yeah, I've been making it check everytime the box is edited which is bad on my behalf. Will update.

d) Making an xray finally
The x-ray that is generated (note: I didn't edit aliases, so we've just got the default character names), and I get garbage. I'm sure Peter is a nice guy, but he really isn't a main character, and certainly he isn't mentioned 65870 times (compared to zero for everyone else!). [I'm sure when I ran this yesterday I got the same result, but with Fred getting all the action]

sqlite> select id, label, count from entity;
id label count
---------- ---------- ----------
0
2 Jacob Marl 0
3 Bob Cratch 0
4 Ghost of C 0
5 The Ghost 0
6 The Ghost 0
7 Mrs. Cratc 0
8 Tiny Tim 0
9 Martha Cra 0
10 Peter Crat 65870
11 Belinda Cr 0
12 Mr. Fezziw 0
13 Mrs. Fezzi 0
14 Ebenezer S 0
15 Dick Wilki 0
16 Fred 0
17 Jacob Marl 0
18 Joe Miller 0
19 Caroline 0
20 Topper 0
21 Belle 0
22 Fan 0
23 Almshouse 0
24 Blind man' 0
25 Humbug 0
sqlite>

- Can you send me the book to try this out with. I could've sworn I got the book parsing down. It works for all my books but maybe I missed something. Or I messed something when making changes for the aliases.

e) editing aliases
I tried adding a single alias ("Scrooge" for the main character) and that had no effect - still Peter Cratchit gets all the action, as in previous case.

- look at d

3) Latest github version in OS X
a) unmodified
I cloned your github and tried that out in OS X - it doesn't work, as I see mentioned in the forum. It would be a shame to not support all OS, especially when the current problem is so minor. You just need to figure out a way to look for external devices in OS X/linux - I suspect something with 'mount' would work.

- Yes, I haven't added support for OS X and Linux. I'm the only dev on this project and I don't have access to either system at the moment. I'm looking into it but this will take some time to add.

b) hacked to workaround problem

4) general comments
a) error reporting
you seem to be swallowing some errors, instead of printing the original python stacktrace. eg. I think I got the "windll" assertion with the 2.0.0 version of the plugin, but I get something more cryptic with the github version (KeyError for 'send_status'). That might make debugging harder..

- Still working on catching all the bugs but I feel like the user doesn't really need to see all the exceptions thrown so I print prettier error messages. Example, if the amazon page isn't found, instead of throwing the httplib exception thrown, I print Amazon page not found. You can see this in the job details.

b) interacting with calibre DB
I had been putting this step off, as I hadn't figured out how to do it. I *think* what you are doing is considered naughty, you shouldn't be modifying the Calibre DB under the covers. I meant to start a thread on how to fix that, hopefully Kovid or other has some ideas.

[I think if a book gets renamed or deleted, you'll be leaving junk scattered around the filesystem. Not a terminal problem, but something to be aware of]

- As far as I can see, editing the calibre db isn't bad. I'm using Calibre's functions to do it so I'm not opening the DB and just modifying directly or anything. See here: https://manual.calibre-ebook.com/gen...l#set-metadata

c) interacting with kindle
As with (b) I hadn't figured this out. I wondered if Kindle driver would need to be modified. Again, I was going to ask the experts.

[what you are currently doing is duplicating code from Kindle driver, but it should be harmless as you'll only be doing what I do by hand currently]

- Yeah I wasn't sure how to do it using Calibre's built in functions. I end up doing it manually (which also explains why I can't support OSX and Linux just yet.. I have no idea how to get to the kindle)

d) modifying ASIN in books directly
If I understand your code you are modifying the MOBI on the Kindle device directly to update the ASIN. This seems odd! FWIW the "Quality Check" plugin can update ASIN as needed, so might be better to just call that to fix books in the Calibre DB directly.

- This was actually quite a complicated decision I made. I wanted to be able to support multiple book formats. Calibre allows you to put multiple formats of the book on your device if you want to. If there's a book with ASIN 123 and it has both azw3 and mobi on the device, the sdr folder for both books will be exactly the same meaning I would have to put 2 x-ray files in the sdr with the same exact name which obviously isn't possible. To get around this, I update the asin on the device to add the format to the end effectively allowing both books to have an x-ray. I don't why anyone would want to have more than one format of a book on a kindle but that's not for me to decide.




All of that being said, I did see your version a few days ago and I saw that your github page states it's not fully functional. Maybe we can merge projects or something. I would definitely love all the help I can get coding (especially considering my issue with other OS'), responding to bug reports and feature requests, and making design decisions. Like I said, I'm the only dev on this project. I did have some help from the other author listed but he helped me with figuring out how to do certain things rather coding. There is a lot of clean up I need to do which is probably what I'll be doing for the next version. PM me if you're interested in merging or something.

[BrentB]

I just installed the plug-in and I get the following error when trying to create the x-ray with my Kindle Voyage plugged in. It is an AZW3 file.
"There is no disk in the drive. Please insert a disk into drive\Device\Harddisk2\DR2." Is it not recognizing my Voyage for some reason? The device shows up in Calibre. I am using Windows 7.

After I clicked on 'Continue' a couple of times, it "seems" to have completed, but the X-Ray option in the menu on my Voyage is still greyed out on the book I was testing. Is there a way to tell in Calibre if the X-Ray was created or not?

[PeterT]

I've seen that behavior from other applications at time, for instance Kobo Desktop; never figured out exactly what causes it, and yes, eventually the click of the continue button succeeds.

[szarroug3]

No there's no way to see it in Calibre but you can check if it's on the device. If it's not, you can try the "send to device" option. I've never tested it with a voyage so I don't know if there's a difference or not. If the send option doesn't send it to the device, you can manually copy it over. You can find the generated x-ray file by right clicking on the book and saying "open containing folder". It will be in the sdr folder in a folder named after the format you created the x-ray file for.

[BrentB]

Strange, but it is working on my work computer (I keep my Calibre library on Dropbox). I wonder if it is some setting in Calibre? I looked and can't find anything.

[szarroug3]

Originally Posted by BrentB

Strange, but it is working on my work computer (I keep my Calibre library on Dropbox). I wonder if it is some setting in Calibre? I looked and can't find anything.

What OS is your work computer? I haven't tested in Windows 7 but I have tested in Windows 8 and 10. Maybe it's a Windows 7 issue?

[fardeenah]

just a quick question, will this x ray creator works well even if i use the [GUI Plugin] Hyphenate This! plugin which add hyphenation? my kindle is not with me, thought u could test it out, thank you

[BrentB]

Originally Posted by szarroug3

What OS is your work computer? I haven't tested in Windows 7 but I have tested in Windows 8 and 10. Maybe it's a Windows 7 issue?

They are both Windows 7. Just tried it on my home computer and now I don't get the error.

[BrentB]

Whoops, I didn't have my Voyage connected when I created the first X-Ray file on my home computer. I tried it with the ereader connected and I did get that error again. But, after clicking on "Try Again" twice, it worked. It did send the X-Ray file to the device.

[szarroug3]

Originally Posted by BrentB

Whoops, I didn't have my Voyage connected when I created the first X-Ray file on my home computer. I tried it with the ereader connected and I did get that error again. But, after clicking on "Try Again" twice, it worked. It did send the X-Ray file to the device.

Awesome! Glad it worked out

[szarroug3]

Added 2.1.1.. No new features but it has some pretty important bug fixes

[Mr.Husky]

Is it possible to add livelib.ru as X-Ray source? Here's a book for example - http://www.livelib.ru/book/1001126888 There's interesting facts, history, plot, critics, etc.

Unfortunately, Amazon/Shelfari doesn't contain any russian book info.

[stoduk]

Naki: this is now working on OS X.

Anyone out there using Linux? The same code should work for that, but it is untested, so if you can try it out that would be useful!

Originally Posted by szarroug3

Ah, that's definitely why it doesn't work for you. There's a lot of code I didn't know how to write for Mac (specifically how to locate the device which is what is failing for you right now). I'd love the help in beta testing but it will take some time before I even get to that stage. I have to check first how much of my code is incompatible and then see how to fix it before we can get to that part. As far as I know, you should be able to get through the "Create X-Ray" function successfully if you don't have "send to device if connected" checked in the preferences. Can you try that? If that works, you should be able to manually copy the x-ray file but you'd have to check the ASIN on the book on the device to make sure it's the same as the ID in the x-ray filename. If it isn't, you can just change the filename so you don't have to change the ASIN in the book. So sorry for the inconvenience!

[szarroug3]

Originally Posted by Mr.Husky

Is it possible to add livelib.ru as X-Ray source? Here's a book for example - http://www.livelib.ru/book/1001126888 There's interesting facts, history, plot, critics, etc.

Unfortunately, Amazon/Shelfari doesn't contain any russian book info.

I've looked at the link you posted and it doesn't seem to have any character or term descriptions. I was reading it translated via Google Translate so maybe I'm missing something. If I'm right, then I'm not sure what exactly you wanted us to put in the x-ray. X-Ray only stores information about characters and terms in the book plus some notable quotes.