[Metadata Source Plugin] Kobo Metadata - Page 9

jason3232 · 06-21-2024, 02:55 PM

It seems if a search only return 1 result e.g. search with ISBN then Kobo will redirect to the book page instead of the search page, maybe that's the reason?

NotSimone · 06-23-2024, 07:48 AM

I've noticed this too and I think the javascript engine isn't fully loading before it parses the page. I'll have a play around with this in the next few days to see what we can do about it, but in the meantime, retry the search until it goes through.

nAhu · 06-27-2024, 03:42 PM

Hi NotSimone

There is a thing I find very cool in the "Kobo Books" plugin ( https://www.mobileread.com/forums/sh...d.php?t=253838 )
It add a value ( kobo: ) in the ids field of calibre with the ending part of the url on the kobo web site.

Example
For "Cooking with Wild Game: Volume 24"
the plug in adds the id : kobo:cooking-with-wild-game-volume-24
which matchs with the link :
https://www.kobo.com/ebook/cooking-w...game-volume-24

Is there any way you can reuse the same thing to match the book in kobo store to get metadata ?

thx for the plugin

ownedbycats · 06-27-2024, 04:37 PM

There's some issues with the kobo: id that makes NotSimone reluctant to use it:

Quote:

Originally Posted by ownedbycats

Personally, I use the kobo: identifiers (https://www.kobo.com/ca/en/ebook/{id}) used by the older Kobo Books plugin. of course it does have its own problems, e.g. the number at the end changing when the book's edition gets updated.

Unfortunately, since the passing of davidfor I've found that plugin failing more and more often, presumably due to site changes, which is why I was looking at this as a replacement (or supplement).

Quote:

Originally Posted by NotSimone

It looks like Kobo 404's the previous edition once they update with a new one so I'm not too keen on using this as an identifier. If anyone comes up with a better one, then I'm happy to use that instead, or alternatively, remove the links entirely.

NotSimone · 07-03-2024, 10:03 AM

I've just pushed a new version which will retry if it detects a failure to defeat the Cloudflare protection up to 10 times. From just playing around, it seems about half the time it takes 0 attempts, and the other half around 3/4. I haven't seen more than 5 yet though.

Give it a try and let me know how it goes

Quote:

Originally Posted by ownedbycats

There's some issues with the kobo: id that makes NotSimone reluctant to use it:

Yep that's pretty much it - the identifier isn't reliable.

josephyw · 07-23-2024, 04:03 AM

Unfortunately, this plugin hasn't worked for me in some time. I am currently on v1.6.3 of this plugin on Calibre 7.15, and I am still getting unyielding 403 errors, like the following:

Code:

Running identify query with parameters: 
{'title': 'Nostalgia', 'authors': ['Mircea Cǎrtǎrescu'], 'identifiers': {}, 'timeout': 30} 
Using plugins: Kobo Books (1, 10, 2), Google (1, 1, 1), Amazon.com (1, 3, 9) 
The log from individual plugins is below 

****************************** Kobo Books (1, 10, 2) ****************************** 
Found 0 results 
Downloading from Kobo Books took 0.17281770706176758 
identify - title: "Nostalgia" authors= "['Mircea Cǎrtǎrescu']"
create_query - title: "Nostalgia"
create_query - after decode title: "Nostalgia"
create_query - title_tokens: "['Nostalgia']"
Querying: https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all
Failed to make identify query: 'https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all'
Traceback (most recent call last):
  File "calibre_plugins.kobobooks.__init__", line 222, in identify
    response = br.open_novisit(query, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden 

********************************************************************************

I am unclear what to do to re-enable the plugin's functionality, as the dev has been responsive for incorporating solutions, but nothing has worked for me with each iteration of the plugin. Does anybody have any ideas to overcome this 403 error? I greatly appreciate and admire what it takes to learn to code and develop something like this.

JSWolf · 07-23-2024, 06:25 AM

I'm not able to download covers from Kobo.

Code:

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0')] 
Failed to download valid cover 
Took 3.3782670497894287 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: The Long Walk, authors: ['Stephen King'], identifiers: {'isbn': '9781501141324'}
KoboMetadata::perform_isbn_search: Getting metadata with isbn: 9781501141324
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No identifier - performing search
KoboMetadata::perform_search: Searching with query: The Long Walk Stephen King
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No search results
KoboMetadata::download_cover: Could not get cover

P.S. I went to the URL in the log in Firefox and then I was able to download the cover.

PeterT · 07-23-2024, 10:52 AM

The problem is the CloudFlare protection on the web site.

Sent from my Pixel 7a using Tapatalk

NotSimone · 07-23-2024, 11:14 PM

Quote:

Originally Posted by josephyw

Unfortunately, this plugin hasn't worked for me in some time. I am currently on v1.6.3 of this plugin on Calibre 7.15, and I am still getting unyielding 403 errors, like the following:

Code:

Running identify query with parameters: 
{'title': 'Nostalgia', 'authors': ['Mircea Cǎrtǎrescu'], 'identifiers': {}, 'timeout': 30} 
Using plugins: Kobo Books (1, 10, 2), Google (1, 1, 1), Amazon.com (1, 3, 9) 
The log from individual plugins is below 

****************************** Kobo Books (1, 10, 2) ****************************** 
Found 0 results 
Downloading from Kobo Books took 0.17281770706176758 
identify - title: "Nostalgia" authors= "['Mircea Cǎrtǎrescu']"
create_query - title: "Nostalgia"
create_query - after decode title: "Nostalgia"
create_query - title_tokens: "['Nostalgia']"
Querying: https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all
Failed to make identify query: 'https://www.kobo.com/search?query=Nostalgia+Mircea+C%C7%8Ert%C7%8Erescu&fcmedia=Book&fclanguages=all'
Traceback (most recent call last):
  File "calibre_plugins.kobobooks.__init__", line 222, in identify
    response = br.open_novisit(query, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "mechanize/_mechanize.py", line 241, in open_novisit
  File "mechanize/_mechanize.py", line 313, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden 

********************************************************************************

I am unclear what to do to re-enable the plugin's functionality, as the dev has been responsive for incorporating solutions, but nothing has worked for me with each iteration of the plugin. Does anybody have any ideas to overcome this 403 error? I greatly appreciate and admire what it takes to learn to code and develop something like this.

The logs look like you are using the older Kobo Books plugin, not this one (Kobo Metadata). Can you try uninstalling that plugin and trying again? Or making sure you select the right one in the metadata providers list.

NotSimone · 07-23-2024, 11:16 PM

Quote:

Originally Posted by JSWolf

I'm not able to download covers from Kobo.

Code:

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0')] 
Failed to download valid cover 
Took 3.3782670497894287 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: The Long Walk, authors: ['Stephen King'], identifiers: {'isbn': '9781501141324'}
KoboMetadata::perform_isbn_search: Getting metadata with isbn: 9781501141324
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=9781501141324&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No identifier - performing search
KoboMetadata::perform_search: Searching with query: The Long Walk Stephen King
KoboMetadata::perform_query: Searching for book with url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - trying again for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_webpage: Could not defeat cloudflare protection - giving up for https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::perform_query: Could not get url: https://www.kobo.com/us/en/search?query=The+Long+Walk+Stephen+King&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::get_cover_url:: No search results
KoboMetadata::download_cover: Could not get cover

P.S. I went to the URL in the log in Firefox and then I was able to download the cover.

Is this consistent or intermittent? Personally the reliability is close to 90% when I use it myself - maybe it needs a wait timer in between attempts or just more attempts before giving up. Not sure what the proper way to do this is but I just flew out on holiday today so this goes into the todo later bucket

JSWolf · 07-24-2024, 04:56 PM

Quote:

Originally Posted by NotSimone

Is this consistent or intermittent? Personally the reliability is close to 90% when I use it myself - maybe it needs a wait timer in between attempts or just more attempts before giving up. Not sure what the proper way to do this is but I just flew out on holiday today so this goes into the todo later bucket

I was away for 8 days recently. I have the cover lookup fail twice before the third time I looked at the log. Those two times, I was able go get the cover from Amazon and Google using the cover lookup. But I got it working when I used the URL in the log with Firefox. So maybe we just have to resign ourselves to using the URL in our browser every so often.

Enjoy your vacation.

NotSimone · 10-28-2024, 01:12 AM

Kobo has updated their bot challenge page and renamed isbn to book id - I've updated the plugin to handle this properly. I'm fully expecting them to randomly go fiddling with the pages again so lmk if this doesn't work properly with any pages.

NotSimone · 10-28-2024, 01:13 AM

Quote:

Originally Posted by JSWolf

I was away for 8 days recently. I have the cover lookup fail twice before the third time I looked at the log. Those two times, I was able go get the cover from Amazon and Google using the cover lookup. But I got it working when I used the URL in the log with Firefox. So maybe we just have to resign ourselves to using the URL in our browser every so often.

Enjoy your vacation.

I've slightly adjusted the bot protection bypass logic so it should be more reliable - let me know if this keeps happening with the new version.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Metadata Source Plugin] Kobo Books	davidfor	Plugins	201	10-17-2023 12:19 PM
[Metadata Source Plugin] BOL.com (DE)	Pr.BarnArt	Plugins	52	06-01-2023 05:50 PM
How do I know which Metadata Source Plugin was called from fetch-ebook-metadata?	eryMpexI	Library Management	6	04-22-2023 09:58 AM
Read a book's metadata in a Metadata source plugin?	J-H	Development	2	03-30-2021 10:08 AM
[Metadata Source Plugin] Empty Plugin? (Fake Identifier)	mneimeyer	Plugins	3	11-11-2019 09:07 PM

06-21-2024, 02:55 PM	#121
jason3232 Junior Member Posts: 2 Karma: 10 Join Date: Sep 2017 Device: Kobo Libra 2, Boox Page	It seems if a search only return 1 result e.g. search with ISBN then Kobo will redirect to the book page instead of the search page, maybe that's the reason?

06-23-2024, 07:48 AM	#122
NotSimone Book Enjoyer Posts: 58 Karma: 704770 Join Date: Sep 2023 Device: Kobo Libra 2	I've noticed this too and I think the javascript engine isn't fully loading before it parses the page. I'll have a play around with this in the next few days to see what we can do about it, but in the meantime, retry the search until it goes through.

06-27-2024, 03:42 PM	#123
nAhu Enthusiast Posts: 33 Karma: 102928 Join Date: Apr 2018 Location: France Device: Kobo Glo (kobo4) / Kobo Libra Colour (kobo11)	Hi NotSimone There is a thing I find very cool in the "Kobo Books" plugin ( https://www.mobileread.com/forums/sh...d.php?t=253838 ) It add a value ( kobo: ) in the ids field of calibre with the ending part of the url on the kobo web site. Example For "Cooking with Wild Game: Volume 24" the plug in adds the id : kobo:cooking-with-wild-game-volume-24 which matchs with the link : https://www.kobo.com/ebook/cooking-w...game-volume-24 Is there any way you can reuse the same thing to match the book in kobo store to get metadata ? thx for the plugin

07-23-2024, 10:52 AM	#128
PeterT Grand Sorcerer Posts: 12,734 Karma: 75000000 Join Date: Nov 2007 Location: Toronto Device: Libra H2O, Libra Colour	The problem is the CloudFlare protection on the web site. Sent from my Pixel 7a using Tapatalk

10-28-2024, 01:12 AM	#132
NotSimone Book Enjoyer Posts: 58 Karma: 704770 Join Date: Sep 2023 Device: Kobo Libra 2	Kobo has updated their bot challenge page and renamed isbn to book id - I've updated the plugin to handle this properly. I'm fully expecting them to randomly go fiddling with the pages again so lmk if this doesn't work properly with any pages.