[Metadata Source Plugin] Kobo Metadata

[NotSimone]

I’ve noticed that cloudflare seems to let certain request through if you’ve done a similar request through the browser already - but only if it’s from the same device. I think there’s some kind of whitelist period after you hit it with a browser, which makes this incredibly annoying to test with actually.

Taking the cookie from a browser and sending along with the calibre plugin request doesn’t seem to work either

Re: Firefox - it wouldn’t be able to fill in the calibre metadata. If all you wanted was the cover, it’s not too difficult to grab it manually from the page with the browser.

[JSWolf]

Originally Posted by NotSimone

I’ve noticed that cloudflare seems to let certain request through if you’ve done a similar request through the browser already - but only if it’s from the same device. I think there’s some kind of whitelist period after you hit it with a browser, which makes this incredibly annoying to test with actually.

Taking the cookie from a browser and sending along with the calibre plugin request doesn’t seem to work either

Re: Firefox - it wouldn’t be able to fill in the calibre metadata. If all you wanted was the cover, it’s not too difficult to grab it manually from the page with the browser.

How about changing the browser identifier to that of Microsoft Edge and see if that works any better.

[NotSimone]

As an update - I've gotten past the cloudflare bot protection with cloudscraper. All it needed was a different javascript interpreter and it started going. Hopefully I'll get a fixed version out in the next few days after I figure out how to get everything to import.

[JSWolf]

Originally Posted by NotSimone

As an update - I've gotten past the cloudflare bot protection with cloudscraper. All it needed was a different javascript interpreter and it started going. Hopefully I'll get a fixed version out in the next few days after I figure out how to get everything to import.

Can you at least give instructions on how to get it working so we can use it until you get it sorted?

[NotSimone]

I just updated with version 1.6.0 which should be able to bypass the cloudflare protection. It involves a decent refactor so let me know if something doesn't work.

And also note - if you recently updated, you may need to wait a little bit for cloudflare to let you in. If it doesn't work, give it a day and try again later.

[JSWolf]

Originally Posted by NotSimone

I just updated with version 1.6.0 which should be able to bypass the cloudflare protection. It involves a decent refactor so let me know if something doesn't work.

And also note - if you recently updated, you may need to wait a little bit for cloudflare to let you in. If it doesn't work, give it a day and try again later.

I'm getting an odd error with the new version.

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36')] 
Failed to download valid cover 
Took 1.3365850448608398 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: Exit Strategy, authors: ['Martha Wells'], identifiers: {'isbn': '9781250185464'}
KoboMetadata::get_cover_url: Getting metadata with isbn: 9781250185464
KoboMetadata::identify: Searching for book with url: https://www.kobo.com/us/en/search?qu...clanguages=all
KoboMetadata::parse_search_page: Detected old search page
KoboMetadata::parse_search_page: Detected old search page
KoboMetadata::parse_search_page: Detected old search page
KoboMetadata::get_cover_url: Searching with query: Exit Strategy Martha Wells
KoboMetadata::identify: Searching for book with url: https://www.kobo.com/us/en/search?qu...clanguages=all
KoboMetadata::parse_search_page: Detected old search page
KoboMetadata::parse_search_page: Detected old search page
KoboMetadata::parse_search_page: Detected old search page
KoboMetadata::get_cover_url:: No search results
KoboMetadata::download_cover: Could not get cover 

********************************************************************************

[nAhu]

Hi,

Also seems to have problems, launching the research but when I want to check the metadata nothing appends, no windows and log seem weird.
Metadata fetch fail with an error "Could not get url: h"

But in the cover search part plugin seems to have the info

#################### Log for Cooking with Wild Game: Volume 24 ####################
Running identify query with parameters: 
{'title': 'Cooking with Wild Game: Volume 24', 'authors': ['EDA'], 'identifiers': {'isbn': '9781718334465', 'kobo': 'cooking-with-wild-game-volume-24'}, 'timeout': 30} 
Using plugins: Kobo Metadata (1, 6, 0) 
The log from individual plugins is below 

****************************** Kobo Metadata (1, 6, 0) ****************************** 
Found 0 results 
Downloading from Kobo Metadata took 3.777388095855713 
KoboMetadata::identify: title: Cooking with Wild Game: Volume 24, authors: ['EDA'], identifiers: {'isbn': '9781718334465', 'kobo': 'cooking-with-wild-game-volume-24'}
KoboMetadata::identify: Getting metadata with isbn: 9781718334465
KoboMetadata::identify: Searching for book with url: https://www.kobo.com/fr/en/search?query=9781718334465&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::identify: Searching with query: Cooking with Wild Game: Volume 24 EDA
KoboMetadata::identify: Searching for book with url: https://www.kobo.com/fr/en/search?query=Cooking+with+Wild+Game%3A+Volume+24+EDA&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::parse_search_page: Detected new search page
KoboMetadata::identify: Looking up metadata with url: h
KoboMetadata::get_webpage: Got exception while opening url: Invalid URL 'h': No scheme supplied. Perhaps you meant https://h?
KoboMetadata::identify: Could not get url: h 

******************************************************************************** 
The identify phase took 3.81 seconds 
The longest time (3.777388) was taken by: Kobo Metadata 
Merging results from different sources 
We have 0 merged results, merging took: 0.00 seconds 
Failed to download metadata for Cooking with Wild Game: Volume 24 

****************************** Kobo Metadata Covers ****************************** 
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0')] 
Downloaded cover: 1400x2056 
Took 3.557833671569824 seconds 
KoboMetadata::download_cover: No cached url found, running identify
KoboMetadata::get_cover_url: title: Cooking with Wild Game: Volume 24, authors: ['EDA'], identifiers: {'isbn': '9781718334465', 'kobo': 'cooking-with-wild-game-volume-24'}
KoboMetadata::get_cover_url: Getting metadata with isbn: 9781718334465
KoboMetadata::identify: Searching for book with url: https://www.kobo.com/fr/en/search?query=9781718334465&fcmedia=Book&pageNumber=1&fclanguages=all
KoboMetadata::parse_book_page: Got title: Cooking with Wild Game: Volume 24
KoboMetadata::parse_book_page: Got authors: ['Eda']
KoboMetadata::parse_book_page: Got series: Cooking with Wild Game
KoboMetadata::parse_book_page: Got series_index: 24
KoboMetadata::parse_book_page: Got publisher: J-Novel Club
KoboMetadata::parse_book_page: Got pubdate: 2024-06-03 00:00:00+00:00
KoboMetadata::parse_book_page: Got isbn: 9781718334465
KoboMetadata::parse_book_page: Got language: English
KoboMetadata::parse_book_page: Got tags: {'Science Fiction & Fantasy', 'Comics Graphic Novels & Manga', 'Fiction & Literature', 'Fantasy', 'Manga'}
KoboMetadata::parse_book_page: Got comments: b'
Things have been exciting recently for the people of the forest’s edge: Shin Ruu has been crowned the victor in the Genos swordsmanship tournament, Lem Dom has earned the right to walk the path toward becoming a hunter, and the smaller clans surrounding the Fa have held their joint festival of the hunt at the start of a new break period.
\n
With all the time off they now have, the clans are now working together to finally show the last holdouts at the forest’s edge how to make delicious food. But will everyone they’re trying to teach appreciate their efforts? Also, what has happened to Myme and Mikel? And perhaps most importantly of all, when a familiar face returns to Genos for the first time in half a year, will things with Vina Ruu work out the way he hopes?
\n
See all this and more in the exciting twenty-fourth volume of Cooking with Wild Game!
\n
 \r\n    '
KoboMetadata::parse_book_page: Got cover: https://cdn.kobo.com/book-images/ca6295db-71c9-42a0-9f1e-7c8b18298a14/cooking-with-wild-game-volume-24.jpg 

********************************************************************************

[NotSimone]

Metadata fetch fail with an error "Could not get url: h"

This should be fixed now with 1.6.1, which I've just uploaded - it was an issue related to picking up metadata when the isbn is already specified.

I'm getting an odd error with the new version.

No idea about this one - I can't seem to replicate. But as part of the version bump, I've added some more logging around it so hopefully we can see what went wrong next time.

[NotSimone]

Ahh actually I found the issue and it should be fixed now in 1.6.2.

[nAhu]

Originally Posted by NotSimone

Ahh actually I found the issue and it should be fixed now in 1.6.2.

Reporting after test, all seems good for me

Thx for the work and sharing

[JSWolf]

Originally Posted by NotSimone

Ahh actually I found the issue and it should be fixed now in 1.6.2.

It's working now to download the cover. Thank you.

What was the issue?

[NotSimone]

Originally Posted by JSWolf

It's working now to download the cover. Thank you.

What was the issue?

It was something related to isbns and specifically cover only searches. Tbh I really should put in unit tests but I haven’t gotten around to figuring out how to do that for calibre plugins yet…

[ownedbycats]

do you mean test versions of the plugin? Most devs just attach them in a reply post and then remove them once it's official released (example)

[NotSimone]

Originally Posted by ownedbycats

do you mean test versions of the plugin? Most devs just attach them in a reply post and then remove them once it's official released (example)

Nah I was referring to some code that you can put in that simulates specific scenarios so you can test your code automatically without trying it manually. Unfortunately I don’t know how to get it to play nice with calibre specific imports though so it’s been stuck on the todo list for a while

[jason3232]

Sometimes the plugin gets the "Please turn on Javasciprt and Cookie" page when searching for a book, is there a fix for it or am I doing something wrong?