11-02-2023, 01:36 AM | #1 |
Emmet Murphy
Posts: 5
Karma: 10
Join Date: Apr 2022
Location: San Francisco, CA, USA
Device: Kindle Oasis
|
WSJ recipe gets "Too Many Requests" error for some articles
The WSJ recipe will download around 70% of the first articles for an issue, but the remaining articles will get only the text "Too Many Requests". Could the explanation be that archive.is rate limits requests, for example allowing only so many requests per minute? (PR 2075 in release 6.29 notes the switch to archive.is: https://github.com/kovidgoyal/calibre/pull/2075)
Assuming that rate limiting is the root cause, is there a way to configure the number of download threads or otherwise slow down Calibre downloads so that it can get all articles for each WSJ issue? |
11-02-2023, 03:23 AM | #2 |
Evangelist
Posts: 489
Karma: 82764
Join Date: May 2021
Device: kindle
|
you can always add delay = 1 # seconds but I don't think the problem can be fixed this way. The rate limit may not be IP specific. And also there are many non-calibre users that use archive.is to read wsj.
Do tell if adding delay to the recipe worked! I actually haven't faced "Too Many Requests" issue, maybe its because of me being on another side of the world or different servers here. Try VPN. |
Advert | |
|
11-02-2023, 03:29 AM | #3 |
Evangelist
Posts: 489
Karma: 82764
Join Date: May 2021
Device: kindle
|
Actually someone can fix the wsj login recipe to decrypt the articles as suggested by kovidgoyal and let the free recipe users use the archive.
https://www.mobileread.com/forums/sh...4&postcount=17 |
12-12-2023, 02:49 AM | #4 | |
Emmet Murphy
Posts: 5
Karma: 10
Join Date: Apr 2022
Location: San Francisco, CA, USA
Device: Kindle Oasis
|
Quote:
|
|
12-18-2023, 06:35 PM | #5 |
Connoisseur
Posts: 64
Karma: 10
Join Date: Dec 2010
Device: Kindle Oasis
|
Hi! Could some kind soul explain to a liberal arts major the steps to add the above mentioned delay to the WSJ recipe?
|
Advert | |
|
12-18-2023, 09:54 PM | #6 |
creator of calibre
Posts: 44,103
Karma: 22669824
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Its already added to the builtin recipe. So just make sure you are using the builtin recipe not a custom one.
|
01-06-2024, 11:40 PM | #7 |
Connoisseur
Posts: 64
Karma: 10
Join Date: Dec 2010
Device: Kindle Oasis
|
I think I am. It says By Covid Goyal. But only about 3/4 of articles are downloaded. In this Saturday's edition, it conks out about halfway through the Review section. Am I doing something wrong?
|
01-09-2024, 02:44 AM | #8 |
Evangelist
Posts: 489
Karma: 82764
Join Date: May 2021
Device: kindle
|
if these articles can be accessed from archive.is, it'll show up in the recipe.
adding delay only fixes "Too Many Requests" problem. |
01-09-2024, 05:08 AM | #9 |
Connoisseur
Posts: 64
Karma: 10
Join Date: Dec 2010
Device: Kindle Oasis
|
The recipe I am using, which I believe is the ‘built-in’ one, downloads the first maybe 75% of articles. The remainder appear as blank, except for a URL…
|
01-09-2024, 06:05 AM | #10 |
Connoisseur
Posts: 64
Karma: 10
Join Date: Dec 2010
Device: Kindle Oasis
|
Should I use the recipe that requires a password (I have one) or the one titled Free?
Thank you. |
01-09-2024, 12:12 PM | #11 |
Evangelist
Posts: 489
Karma: 82764
Join Date: May 2021
Device: kindle
|
technically both the recipes are the same.
|
01-14-2024, 07:07 PM | #12 |
Connoisseur
Posts: 64
Karma: 10
Join Date: Dec 2010
Device: Kindle Oasis
|
For the weekend WSJ the first 69 articles appear correctly. The last correctly loaded article is from the Review section. It is a review of a book about Madame de Pompadour. For the next article, all that appears on my kindle is "This article was downloaded by Calibre from [article URL]." Going forward some articles appear but others produce the "downloaded by Calibre" message
Is this the case for others? Am I doing something wrong? Any fix? Thank you as always to the more knowledgeable members of this community who help our the rest of us! Last edited by mjfriedman; 01-14-2024 at 09:43 PM. |
01-14-2024, 11:10 PM | #13 |
Evangelist
Posts: 489
Karma: 82764
Join Date: May 2021
Device: kindle
|
|
01-15-2024, 09:52 AM | #14 |
Connoisseur
Posts: 64
Karma: 10
Join Date: Dec 2010
Device: Kindle Oasis
|
So it seems this is known issue with no fix at present. Grateful though, to get 75-80% of the paper....
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Calibre Server: 2 Feature Requests/Ideas... "tag cloud"+"word cloud" | wjBLqf-_9 | Server | 13 | 11-11-2019 12:03 PM |
Wall Street Journal (WSJ) recipe - "stuck on 9/5" | stualden | Recipes | 9 | 09-21-2018 09:20 AM |
BBC recipe "connection reset by peer" error | Danfengxinlan | Recipes | 0 | 09-16-2015 12:29 AM |
Cracked.com recipe skips "blog" articles | damien18 | Recipes | 3 | 10-15-2013 10:37 PM |
Error "Could not fetch link..." while creating a basic recipe | cram1010 | Recipes | 4 | 05-05-2012 07:13 AM |