[Metadata Source Plugin] Kobo Metadata

[jason3232]

It seems if a search only return 1 result e.g. search with ISBN then Kobo will redirect to the book page instead of the search page, maybe that's the reason?

[NotSimone]

I've noticed this too and I think the javascript engine isn't fully loading before it parses the page. I'll have a play around with this in the next few days to see what we can do about it, but in the meantime, retry the search until it goes through.

[nAhu]

Hi NotSimone

There is a thing I find very cool in the "Kobo Books" plugin ( https://www.mobileread.com/forums/sh...d.php?t=253838 )
It add a value ( kobo: ) in the ids field of calibre with the ending part of the url on the kobo web site.

Example
For "Cooking with Wild Game: Volume 24"
the plug in adds the id : kobo:cooking-with-wild-game-volume-24
which matchs with the link :
https://www.kobo.com/ebook/cooking-w...game-volume-24

Is there any way you can reuse the same thing to match the book in kobo store to get metadata ?

thx for the plugin

[ownedbycats]

There's some issues with the kobo: id that makes NotSimone reluctant to use it:

Originally Posted by ownedbycats

Personally, I use the kobo: identifiers (https://www.kobo.com/ca/en/ebook/{id}) used by the older Kobo Books plugin. of course it does have its own problems, e.g. the number at the end changing when the book's edition gets updated.

Unfortunately, since the passing of davidfor I've found that plugin failing more and more often, presumably due to site changes, which is why I was looking at this as a replacement (or supplement).

Originally Posted by NotSimone

It looks like Kobo 404's the previous edition once they update with a new one so I'm not too keen on using this as an identifier. If anyone comes up with a better one, then I'm happy to use that instead, or alternatively, remove the links entirely.

[NotSimone]

I've just pushed a new version which will retry if it detects a failure to defeat the Cloudflare protection up to 10 times. From just playing around, it seems about half the time it takes 0 attempts, and the other half around 3/4. I haven't seen more than 5 yet though.

Give it a try and let me know how it goes

Originally Posted by ownedbycats

There's some issues with the kobo: id that makes NotSimone reluctant to use it:

Yep that's pretty much it - the identifier isn't reliable.

[josephyw]

Unfortunately, this plugin hasn't worked for me in some time. I am currently on v1.6.3 of this plugin on Calibre 7.15, and I am still getting unyielding 403 errors, like the following:

Running identify query with parameters: 
{'title': 'Nostalgia', 'authors': ['Mircea Cǎrtǎrescu'], 'identifiers': {}, 'timeout': 30} 
Using plugins: Kobo Books (1, 10, 2), Google (1, 1, 1), Amazon.com (1, 3, 9) 
The log from individual plugins is below 

****************************** Kobo Books (1, 10, 2) ****************************** 
Found 0 results 
Downloading from Kobo Books took 0.17281770706176758 
identify - title: "Nostalgia" authors= "['Mircea Cǎrtǎrescu']"
create_query - title: "Nostalgia"
create_query - after decode title: "Nostalgia"
create_query - title_tokens: "['Nostalgia']"
Querying: https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all
Failed to make identify query: 'https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all'
Traceback (most recent call last):
  File "calibre_plugins.kobobooks.__init__", line 222, in identify
    response = br.open_novisit(query, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden 

********************************************************************************

I am unclear what to do to re-enable the plugin's functionality, as the dev has been responsive for incorporating solutions, but nothing has worked for me with each iteration of the plugin. Does anybody have any ideas to overcome this 403 error? I greatly appreciate and admire what it takes to learn to code and develop something like this.

[JSWolf]

I'm not able to download covers from Kobo.

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0')] 
Failed to download valid cover 
Took 3.3782670497894287 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: The Long Walk, authors: ['Stephen King'], identifiers: {'isbn': '9781501141324'}
KoboMetadata::perform_isbn_search: Getting metadata with isbn: 9781501141324
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No identifier - performing search
KoboMetadata::perform_search: Searching with query: The Long Walk Stephen King
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No search results
KoboMetadata::download_cover: Could not get cover

P.S. I went to the URL in the log in Firefox and then I was able to download the cover.

[PeterT]

The problem is the CloudFlare protection on the web site.

Sent from my Pixel 7a using Tapatalk

[NotSimone]

Originally Posted by josephyw

Unfortunately, this plugin hasn't worked for me in some time. I am currently on v1.6.3 of this plugin on Calibre 7.15, and I am still getting unyielding 403 errors, like the following:

Running identify query with parameters: 
{'title': 'Nostalgia', 'authors': ['Mircea Cǎrtǎrescu'], 'identifiers': {}, 'timeout': 30} 
Using plugins: Kobo Books (1, 10, 2), Google (1, 1, 1), Amazon.com (1, 3, 9) 
The log from individual plugins is below 

****************************** Kobo Books (1, 10, 2) ****************************** 
Found 0 results 
Downloading from Kobo Books took 0.17281770706176758 
identify - title: "Nostalgia" authors= "['Mircea Cǎrtǎrescu']"
create_query - title: "Nostalgia"
create_query - after decode title: "Nostalgia"
create_query - title_tokens: "['Nostalgia']"
Querying: https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all
Failed to make identify query: 'https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all'
Traceback (most recent call last):
  File "calibre_plugins.kobobooks.__init__", line 222, in identify
    response = br.open_novisit(query, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden 

********************************************************************************

I am unclear what to do to re-enable the plugin's functionality, as the dev has been responsive for incorporating solutions, but nothing has worked for me with each iteration of the plugin. Does anybody have any ideas to overcome this 403 error? I greatly appreciate and admire what it takes to learn to code and develop something like this.

The logs look like you are using the older Kobo Books plugin, not this one (Kobo Metadata). Can you try uninstalling that plugin and trying again? Or making sure you select the right one in the metadata providers list.

[NotSimone]

Originally Posted by JSWolf

I'm not able to download covers from Kobo.

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0')] 
Failed to download valid cover 
Took 3.3782670497894287 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: The Long Walk, authors: ['Stephen King'], identifiers: {'isbn': '9781501141324'}
KoboMetadata::perform_isbn_search: Getting metadata with isbn: 9781501141324
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No identifier - performing search
KoboMetadata::perform_search: Searching with query: The Long Walk Stephen King
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No search results
KoboMetadata::download_cover: Could not get cover

P.S. I went to the URL in the log in Firefox and then I was able to download the cover.

Is this consistent or intermittent? Personally the reliability is close to 90% when I use it myself - maybe it needs a wait timer in between attempts or just more attempts before giving up. Not sure what the proper way to do this is but I just flew out on holiday today so this goes into the todo later bucket

[JSWolf]

Originally Posted by NotSimone

Is this consistent or intermittent? Personally the reliability is close to 90% when I use it myself - maybe it needs a wait timer in between attempts or just more attempts before giving up. Not sure what the proper way to do this is but I just flew out on holiday today so this goes into the todo later bucket

I was away for 8 days recently. I have the cover lookup fail twice before the third time I looked at the log. Those two times, I was able go get the cover from Amazon and Google using the cover lookup. But I got it working when I used the URL in the log with Firefox. So maybe we just have to resign ourselves to using the URL in our browser every so often.

Enjoy your vacation.

[NotSimone]

Kobo has updated their bot challenge page and renamed isbn to book id - I've updated the plugin to handle this properly. I'm fully expecting them to randomly go fiddling with the pages again so lmk if this doesn't work properly with any pages.

[NotSimone]

Originally Posted by JSWolf

I was away for 8 days recently. I have the cover lookup fail twice before the third time I looked at the log. Those two times, I was able go get the cover from Amazon and Google using the cover lookup. But I got it working when I used the URL in the log with Firefox. So maybe we just have to resign ourselves to using the URL in our browser every so often.

Enjoy your vacation.

I've slightly adjusted the bot protection bypass logic so it should be more reliable - let me know if this keeps happening with the new version.

[josephyw]

I'm just here to say that I have found the covers search (and, to a further extent, high resolution covers) far more reliable since the upgrade to 1.7.0. Thank you!