[GUI Plugin] Goodreads Sync

[kiwidude]

This plugin allows you to synchronise your book collection in Calibre with your shelves on a Goodreads.com account.

Goodreads targets the social aspects of sharing your book lists with friends and family. It offers various services such as reviews, book clubs, discussions and recommendations. You can maintain shelves of books you have read, want to read, are currently reading or others of your own design.

In some cases this is information that you also want to keep track of within Calibre using tags or custom columns. Typical examples of usage of this plugin are:

• Adding a book that you have just imported to Calibre to your 'to-read' shelf on Goodreads
• Updating a Calibre custom column to indicate that you have read a book when syncing from your 'read' shelf on Goodreads
• Multiple actions can be applied. e.g. in the example above you might also populate a 'Read Date' custom column and remove a 'To Read' tag from the book in Calibre.
• Using your own genre shelves in Goodreads like science-fiction, romance etc to map to Calibre tags or a custom genre column of your choice.

Main Features:

• Add or remove books to one or more Goodreads shelves from a selection within Calibre
• Syncing the contents from one or more Goodreads shelves into Calibre, performing customisable actions for each shelf
• Configure actions to be performed for when adding books to a shelf or syncing from Goodreads, such as updating tags, custom columns.
• You can also synchronise your rating, date read and or review text custom column with your Goodreads review.
• Ability to download tags for your books based on the shelves they are on. You can customise which Calibre tags if any each shelf name will map to and use a custom genre column instead of tags if desired.
• Ability to upload to shelves for your books based on the tags they have been given. Applies the to the same Calibre column and shelf mappings as for the download tags feature above.
• Link your Calibre books to a Goodreads equivalent giving you right-click access to reviews or other information for that book
• Ability to switch editions for a linked Goodreads book
• Ability to create Empty Books in Calibre for books you sync from a Goodreads shelf
• Optionally update the ISBN in Calibre to match the edition on your Goodreads shelf
• Supports multiple Goodreads user accounts if required for users sharing a single operating system account
• Help file web page available from the configuration dialog or plugin menu

Special Notes:

• Help Documentation
• Change History
• Please don't abuse this plugin by using it to put many hundreds or thousands of books on your shelves. Goodreads T&C require a very limited threshold on the number of API calls per minute. Worst case they will block the plugin and that ruins it for everyone. You can use CSV export/import to Goodreads for large volumes if you need it. Refer to the help page within the plugin for the full details.
• Requires Calibre v0.8.18 or later.

Paypal Donations:

• If you find this or any of my other plugins useful please feel free to show your appreciation. I have spent many hundreds of unpaid hours in their development and support so any encouragement for me to continue is appreciated!
  

[innowen]

Huzzah for Kiwidude,

Thanks for putting in the time and effort to code this for us. I'm sure it'll be a hit.

/innowen

[pur]

Yay!! i was eagerly waiting for this one. Thank You so much KiwiDude.

[Qylie]

thanks for all the hard work on this plug-in!

[msr]

Very interesting. Seems to work pretty well so far. A few comments.

1. I have more than 100 shelves in Goodreads, but the plugin only retrieves the first 100. I suspect this is a Goodreads API issue as opposed to an issue with your plugin, but it's something to look into (at this point I don't foresee personally trying to access any past the first few anyway for my own purposes, but it's something others may need to do).

2. Currently one can add a set of books to a Goodreads shelf, or pull all of the books from a shelf and add them to Calibre (and/or add tags or similar options to the data in Calibre. Once a book is linked, I'd like to see more book<->book operations (as opposed to shelf operations). For example, syncing my rating of a book in calibre with the book in Goodreads (or visa-versa). Or creating tags in calibre for a book that match all of the shelves it's on in Goodreads (that is, rather than pulling all books from a shelf, pull all shelves for a book).

Anyway, these are more long-term wishlist (and API allowing) issues. Still a very nice plugin as is.

[kiwidude]

Originally Posted by msr

Very interesting. Seems to work pretty well so far. A few comments.

1. I have more than 100 shelves in Goodreads, but the plugin only retrieves the first 100. I suspect this is a Goodreads API issue as opposed to an issue with your plugin, but it's something to look into (at this point I don't foresee personally trying to access any past the first few anyway for my own purposes, but it's something others may need to do).

More than 100 shelves? Why do you have so many? I am genuinely intrigued?

As for the plugin limiting to 100, that was my own threshold, not a limitation of the API. I could not think of a reason as to why someone would have anywhere near that many, and already someone just has I will remove the threshold in my next release.

2. Currently one can add a set of books to a Goodreads shelf, or pull all of the books from a shelf and add them to Calibre (and/or add tags or similar options to the data in Calibre. Once a book is linked, I'd like to see more book<->book operations (as opposed to shelf operations). For example, syncing my rating of a book in calibre with the book in Goodreads (or visa-versa).

Yes I confess to having had similar thoughts in the past in using this plugin as a way to update metadata, though obviously that overlaps with the metadata download plugins. Pulling data from the book such as a rating is no big deal technically, it is just how to fit it into the UI that I haven't given thought to yet. Suggestions welcomed - particularly as some users would I am sure potentially want to use this as an alternative to the "near-random sourced metadata" that hitting Ctrl+D provides. It opens the same can of worms that some of us have requested more control over with normal metadata download - specifically allowing the ability to choose which fields get overwritten, quite possibly on a book by book basis. You will not "always" want the value for a field from Goodreads if you have already sourced a better value yourself from elsewhere.

As for pushing a rating up to Goodreads, I had someone else mention that as well as I believe that comes under the banner of creating a review? Certainly the API supports that, again it is just the question of the UI design.

Or creating tags in calibre for a book that match all of the shelves it's on in Goodreads (that is, rather than pulling all books from a shelf, pull all shelves for a book).

This sounds a fairly unique requirement. The data is available in the API for every book as to what shelves you have it on but I don't make use of it. What exact example scenario do you have where this would be useful?

[msr]

Originally Posted by kiwidude

More than 100 shelves? Why do you have so many? I am genuinely intrigued?

On Goodreads, shelves are functionally equivalent to tags (the name sometimes throws people off, but they're really just tags). I've gradually taken to using more and more tags for my books (a few are status based, such as reading, own, or to-buy; while most represent either genre and/or topic), so the numbers have gradually grown. I know of people on Goodreads who have hundreds of shelves (I have maybe 120 at this point).

Originally Posted by kiwidude

This sounds a fairly unique requirement. The data is available in the API for every book as to what shelves you have it on but I don't make use of it. What exact example scenario do you have where this would be useful?

Actually, I don't think it is at all (unique) and strikes me as one of the more common things someone might want to do. If you view the shelves as tags and you've already spent a lot of time tagging books in Goodreads, wouldn't you want to use those same tags in calibre?

Just as an example, a number of months ago I got a free ebook copy of Kim Harrison's Dead Witch Walking, which I happened to have already read. It was already on Goodreads with the following "shelves": dark-fantasy, fantasy, fiction, horror, low-fantasy, paranormal-romance, read, romance, urban-fantasy, vampires

It'd be nice to link the ebook to my record already on Goodreads and automatically create/populate all of those tags in Calibre since I've already gone through the effort of creating my own tag structure. I currently do not have a 1:1 correspondence between my Goodreads "tags" (i.e. shelves) and my Calibre tags (I have way fewer Calibre tags), and I waffle on whether I even want this, but if I do decide to go down that path, your plug-in could be the easiest way to achieve it.

I personally have a lot less interest in the community meta-data, but others live and die by it.

[kiwidude]

Thanks for posting the further detail on this. I'm pretty new to Goodreads myself so I don't know exactly how people use it outside of the status type shelves.

Just so I understand - these 120 shelves you have - you created each of them and they only contain books you have put on them?

I am familiar with the shelves as tags concept - in fact the Goodreads metadata plugin I wrote uses exactly this to populate tags in Calibre with a mapping between. However it uses the "popular shelves" for the book displayed on the book page, so a shelf called "science-fiction" or whatever. I had just assumed now seemingly incorrectly that "science-fiction" was some sort of communal shelf name. Is it instead the case that each user has had to create their own "science-fiction" shelf and Goodreads is just recognising the same shelf name from individual users and consolidating the results?

So my Goodreads metadata plugin is an attempt to get the "popular" shelves for a book as tags, whereas what you are asking for is to allow you to use your own shelves for a book as tags, right?

I can see that being useful for people like yourself who have made the effort on Goodreads (compared to others like myself who just maintain a tag in Calibre). I agree on not necessarily wanting a 1:1 mapping with Calibre tags, you would probably want a customisable version of the mapping approach I did for the metadata plugin. It would default to listing all your shelf names for a 1:1, then you would delete from the mapping list any shelves you did not want as tags, and optionally assign Calibre specific tag names to ones you want translated.

[msr]

Originally Posted by kiwidude

Thanks for posting the further detail on this. I'm pretty new to Goodreads myself so I don't know exactly how people use it outside of the status type shelves.

I'm a bit of a Goodreads junkie and fairly active on the site, so have a bit of a different perspective.

Just so I understand - these 120 shelves you have - you created each of them and they only contain books you have put on them?

Yes

I am familiar with the shelves as tags concept - in fact the Goodreads metadata plugin I wrote uses exactly this to populate tags in Calibre with a mapping between. However it uses the "popular shelves" for the book displayed on the book page, so a shelf called "science-fiction" or whatever. I had just assumed now seemingly incorrectly that "science-fiction" was some sort of communal shelf name. Is it instead the case that each user has had to create their own "science-fiction" shelf and Goodreads is just recognising the same shelf name from individual users and consolidating the results?

Yes, exactly. This is why there are so many shelves of very similar (in fact virtually synonymous) names. Some people prefer "science-fiction", others prefer "sci-fi", others "scifi", others "sf". Some lump fantasy in with SF, some separate it (leading to "sci-fi-fantasy", "scifi-fantasy", "sff", "fantasy-scifi", etc.). Then there are spelling variants due to country (e.g., favorite vs. favourite) or plurality (e.g., "ebook" and "ebooks" are both in the top 50 in usage; "e-book" and "e-books" are also in the top 100). Goodreads users appear to have designated over 426k tags (not all of which are actually in current use).

If you go to the shelves listed by popularity on Goodreads you can see a lot of these. For example: "2010", "read-in-2010", "read-2010", and "2010-reads" are all in the top 75 or so, yet are essentially the same shelf. Goodreads doesn't designate specific shelf names (other than read, to-read, currently-reading...and perhaps wishlist?) because they want users to have the flexibility to choose what they will. However, they are working on a method for synonymizing certain names in the background since the huge numbers of synonyms does detract from certain obvious sorts of meta-data compilation.

So my Goodreads metadata plugin is an attempt to get the "popular" shelves for a book as tags, whereas what you are asking for is to allow you to use your own shelves for a book as tags, right?

Exactly

I can see that being useful for people like yourself who have made the effort on Goodreads (compared to others like myself who just maintain a tag in Calibre). I agree on not necessarily wanting a 1:1 mapping with Calibre tags, you would probably want a customisable version of the mapping approach I did for the metadata plugin. It would default to listing all your shelf names for a 1:1, then you would delete from the mapping list any shelves you did not want as tags, and optionally assign Calibre specific tag names to ones you want translated.

That would certainly be a reasonable approach. Thanks for considering this.

[kiwidude]

I've had a few further thoughts on your suggestions. Firstly pulling rating information. I will hold off doing any kind of metadata updating until Kovid finishes his changes to that API. I don't know any details of what he has planned for it, but it is "possible" that he may allow for a more granular merging of data such that you could choose to get only the rating. What I do know for sure is that as he has a new identifiers table I can make use of I will be changing where this plugin currently stores the Goodreads Id so that both the Sync and the Metadata Download plugins can use it. As I mentioned on the other thread this will mean that when you do Ctrl+D the metadata download plugin can use the linked Goodreads Id to read metadata from the specific edition you have linked. So that would allow you to be sure you got just the right rating.

Sending a rating means creating or updating a review. That is something I may consider in future. For now you can just use the "View linked book" option and do it on the Goodreads website.

I've already removed the restriction on # shelves downloaded. I didn't have a limit of 100, I was just only downloading one "page" of your shelves. So it sounds like there are 100 shelves per "page". My change means I now download all "pages" if there are more than one. I have also added a "Display all" and "Display none" right-click menu options on the shelves list in the configuration dialog, to make it faster to decide which will appear on your menus. I hate to imagine what 120 shelves must look like as a submenu...

I will also add a "Create shelf" icon on that same dialog as someone else requested it as well.

As far as getting tags from shelves is concerned, it would be a little bit quirky. The only way via the API that I can get what shelves a book is on is via a call that gets all the books for a shelf. So this currently takes place when you choose to "Sync". What this means is that you won't be able to right-click on a book and say "Download tags from Goodreads" or whatever. Instead it would have to be an option that gets applied in the background as part of syncing from a shelf. While I do also read the books on a shelf when you add/remove I think it is less confusing if this tag updating would only take place for "sync" actions.

So lets say that you know all your books live on your status shelves of "read", "to-read" or "currently-reading" and perhaps another called "wishlist" or whatever. These are arguably the shelves you will most often sync from or update to from Calibre. To add tags for a Calibre book based on your Goodreads shelves, it will have to exist on one of those shelves listed (or whatever other shelves you are willing to sync from). If you have a book just on your "science-fiction" shelf and don't sync from that, you won't be able to get that as a tag for Calibre.

Now that makes it sound like a background action a bit like overwriting the ISBN is. So I think I would have a checkbox option on the configuration dialog of "Create Calibre tags for my shelves when syncing" with a button next to it popping up a dialog allowing you to modify the shelf->tag mappings as mentioned in my previous post. I will also add a "shelves" column to the sync dialog so you can see what of your shelves a book is on.

What this will mean is that every time you sync with that checkbox turned on your tags will potentially be added to for all of the books on that shelf. Tags are never duplicated, but if you had manually deleted a tag put there because you didn't want it in Calibre this action will keep putting it back until you either modify the shelves the book is on in Goodreads or edit your shelf/tag mappings. I don't really want to put the checkbox on the sync dialog itself - I think this is something you choose to commit to in entirety or not at all

The final implication of the above is that I only allow you to sync from a shelf which has at least one action setup for it. So say for instance you have a "to-read" shelf that has lots of your books on it that you want to get tags for. You will have to setup a sync action for that shelf to set some tag or custom column. If that sounds too limiting in that you cannot think of an action you would want for that shelf in Calibre, I could add a new dummy action type to that dropdown which currently says "Add value to column" and "Remove value from column".

Anyways, those are my thoughts. Given the constraints above about it only being able to update books that exist on a shelf you sync from etc is it still worth doing, or is it so unintuitive we bin the whole idea?

[ViktorShchedrin]

thank YOU very much

[sdow1]

I'm generally thoroughly enjoying this new plugin, but I'm getting the following error message when I try to sync my "to-read" shelf from goodreads:

calibre, version 0.7.47
ERROR: Unhandled exception: <b>TypeError</b>:QTableWidgetItem(): arguments did not match any overloaded call:
  overload 1: argument 1 has unexpected type 'NoneType'
  overload 2: argument 1 has unexpected type 'NoneType'
  overload 3: argument 1 has unexpected type 'NoneType'
  overload 4: argument 1 has unexpected type 'NoneType'

Traceback (most recent call last):
  File "/Users/sam/Library/Preferences/calibre/plugins/Goodreads Sync.zip/goodreads_sync_code.py", line 2706, in sync_shelf
    shelf_name, goodreads_shelf_books, self.calibre_searcher)
  File "/Users/sam/Library/Preferences/calibre/plugins/Goodreads Sync.zip/goodreads_sync_code.py", line 1177, in __init__
    self.summary_table.populate_table(self.goodreads_books)
  File "/Users/sam/Library/Preferences/calibre/plugins/Goodreads Sync.zip/goodreads_sync_code.py", line 1065, in populate_table
    self.populate_table_row(row, book)
  File "/Users/sam/Library/Preferences/calibre/plugins/Goodreads Sync.zip/goodreads_sync_code.py", line 1086, in populate_table_row
    self.setItem(row, 2, ReadOnlyTableWidgetItem(goodreads_book['goodreads_author']))
  File "/Users/sam/Library/Preferences/calibre/plugins/Goodreads Sync.zip/goodreads_sync_code.py", line 162, in __init__
    QTableWidgetItem.__init__(self, text)
TypeError: QTableWidgetItem(): arguments did not match any overloaded call:
  overload 1: argument 1 has unexpected type 'NoneType'
  overload 2: argument 1 has unexpected type 'NoneType'
  overload 3: argument 1 has unexpected type 'NoneType'
  overload 4: argument 1 has unexpected type 'NoneType'

All of my other shelves (read, abandoned, currently reading, etc.) are syncing fine.

Thanks for any help, and thanks for developing this plugin!! Other than my problem above, I think it's fantastic.

[kiwidude]

Originally Posted by sdow1

I'm generally thoroughly enjoying this new plugin, but I'm getting the following error message when I try to sync my "to-read" shelf from goodreads:

I haven't seen that one before. Must be something to do with one of the books on that shelf. Send me a PM with your Goodreads id (just copy the hyperlink for when you click on your name on the home page) so I can test with the contents of your shelf.

[msr]

Originally Posted by kiwidude

As far as getting tags from shelves is concerned, it would be a little bit quirky. The only way via the API that I can get what shelves a book is on is via a call that gets all the books for a shelf.

Really? Hmmm. I'm pretty sure that's wrong. I worked with the API a bit once a year or so ago when trying to help someone with an Android app and I thought there was another way. Did you look at the review/show_by_user_and_book.xml API call? Given a user_id (which you obviously need) and a book_id (which you get from your linking) you get a bunch of information on that book including (buried in the xml, toward the bottom) all of the shelves that user has put the book on (I'm looking at it now to confirm that my memory wasn't wrong). This works even if a book hasn't been reviewed by the user (it just has to be on one of their shelves).

You could also use this same call to fetch the user's rating for a book from Goodreads if you wanted to add that rating to Calibre, although not as a dual-way sync since it's a read only call.

Or am I missing something? If you have to do it by looking at an entire shelf, I agree that would not be worthwhile.

[kiwidude]

Ahhh... quite right you are, thanks for that. I hadn't seen that API call, just the others that I use currently. That would make rather more sense from a user perspective in terms of being able to select books and choosing to update their tags. The downside is the volume of additional API calls but it's not as if you should be regularly overwriting your tags.