12-15-2021, 05:12 AM | #1 |
Member
Posts: 12
Karma: 10
Join Date: Mar 2015
Device: Kindle 3
|
Does calibre support retry-after http headers ?
Hi,
A query please. I have been trying to download metadata and frequently get HTTP error 429 'Too Many requests'. My reading implies that when the 429 is supplied sometimes it gives a 'retry-after' response which indicates when service can resume (I don't know if it's Amazon or Google but they definately limit requests in an automated fashion). Is there any possibility, that an interval could be configured whenever a 429 is the response to allow for retries that applies to all plugins (on each request) or is this a can of worms ? Thanks in advance https://developer.mozilla.org/en-US/...TTP/Status/429 |
12-15-2021, 05:15 AM | #2 |
creator of calibre
Posts: 44,167
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
It's a can of worms. That's a bot prevention measure and it will be from google not amazon. amazon uses captchas.
|
Advert | |
|
12-15-2021, 06:19 AM | #3 |
Member
Posts: 12
Karma: 10
Join Date: Mar 2015
Device: Kindle 3
|
Thanks for replying. I appreciate it. I just wondered if there is anything that can be done to reduce the impact such as rate limiting or sleeping using celery or something similar.
Example here: https://stackoverflow.com/a/50294531 I apologise in advance if this is an example of a newb teaching you to suck eggs. Just trying to see if the metadata retrieval process can be improved. |
12-15-2021, 06:45 AM | #4 |
Member
Posts: 12
Karma: 10
Join Date: Mar 2015
Device: Kindle 3
|
this page details some options for backing off, and using rate limiting, would any of that help do you think ?
https://stackoverflow.com/questions/...equests-python |
12-15-2021, 08:05 AM | #5 |
creator of calibre
Posts: 44,167
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
It's already rate limited. Google simply blocks you once you do too many requsts from an IP regardless of the rate
|
Advert | |
|
12-15-2021, 08:29 AM | #6 |
Member
Posts: 12
Karma: 10
Join Date: Mar 2015
Device: Kindle 3
|
If that is the case is is possible to batch them, or limit the number of metadata requests per batch ? If you start getting 429 responses, you may as well just cancel the remainder of the requests ?
Is it possible to show where in the code this is handled, just so that I can familiarise myself with it without wasting your time ? This is what google says on it. https://developers.google.com/calend..._many_requests cheers Simon |
12-15-2021, 11:40 AM | #7 |
creator of calibre
Posts: 44,167
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
It doesnt use an API, it scrapes the website. And there is basically no robust way to do bulk scraping from a single IP address, the server will always win. You just have to limit your queries to no more than a handful a day. And if you want to read the code look in the metadata/sources folder.
|
Tags |
metadata, retry-after, too many requests |
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Auto Retry if Scheduled download fail!!! | jardenman26 | Recipes | 3 | 06-21-2018 05:28 AM |
Auto Retry if Scheduled download fails | stevesout | Calibre | 1 | 03-14-2018 11:26 PM |
Minor UI improvement - retry button | whismerhill | Calibre | 7 | 05-18-2016 01:01 AM |
Retry Job? | truth1ness | Calibre | 1 | 04-01-2015 07:26 PM |
Calibre not removing headers in mobi conversion | kim- | Calibre | 2 | 08-04-2009 11:37 AM |