![]() |
#46 |
Member
![]() Posts: 11
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
|
I hate to say this, but maybe its time to figure out a way to create a site that images amazon without the captcha for the exclusive use of bots to use. MAYBE it is possible to use Wayback machine for that. A friend of mine data mined a site and create a mirror image of the original site. would just require a large storage of at least a TB or more to mirror. Just a thought.
Last edited by Oddball; 02-26-2017 at 06:16 PM. |
![]() |
![]() |
![]() |
#47 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 12,534
Karma: 74358018
Join Date: Nov 2007
Location: Toronto
Device: Libra H2O, Libra Colour
|
I seriously doubt Amazon would allow that
|
![]() |
![]() |
Advert | |
|
![]() |
#48 |
Member
![]() Posts: 11
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
|
well truthfully they are already and have been apart of Archive.org since 2000 at least.....it is the one way i find info on some eBook they have discontinued.
http://amazon.com Saved 52,539 times between December 12, 1998 and February 26, 2017. |
![]() |
![]() |
![]() |
#49 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 12,534
Karma: 74358018
Join Date: Nov 2007
Location: Toronto
Device: Libra H2O, Libra Colour
|
Fine go and make a metadata plugin to access archive.org's copy of Amazon then
|
![]() |
![]() |
![]() |
#50 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,230
Karma: 23446430
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
It's not quite as simple as using a mirror. The problem is metadata plugins require a way to search for title/author/isbn/etc. So in addition to a mirror you'd need to implement search as well.
So directly using something like archive.org is not possible. What one can do is probably use google's sitesearch for archive.org. Though IIRC gogole does not like bots using their search either, so you'd likely run into the same problems as with amazon, eventually. |
![]() |
![]() |
Advert | |
|
![]() |
#51 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,637
Karma: 85000001
Join Date: Nov 2011
Location: Tampa Bay, Florida
Device: Kindles
|
In my personal experience it seems that Amazon doesn't bother users again for quite a while once they answer a captcha, as long as cookies are maintained.
Perhaps a solution would be to present the captcha to the user for solution, proving the non-botness of the requester along with preserving the Amazon cookies in a more permanent fashion. ETA: The function of making a request to Amazon and having the user solve the captcha could be a UI plugin, separate from the Amazon metadata plugin, as long as they share cookies. Last edited by jhowell; 02-28-2017 at 12:25 PM. |
![]() |
![]() |
![]() |
#52 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,230
Karma: 23446430
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
That's awfully fiddly, and probably if amazon detects that bots resume operations after answering a captcha on the same IP it will escalate to banning the IP with HTTP 503 or the like. After all, it is actually fairly trivial to solve captchas programmatically, the problem with doing that is that the next step is getting the IP banned, at least that is how I recall most bot detection algorithms work. A bit of googling suggests that is indeed how they work.
|
![]() |
![]() |
![]() |
#53 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 6,637
Karma: 85000001
Join Date: Nov 2011
Location: Tampa Bay, Florida
Device: Kindles
|
I definitely do not want my IP address banned by Amazon. It is too bad that they are taking these steps to block programmatic access to their site.
|
![]() |
![]() |
![]() |
#54 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,230
Karma: 23446430
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Looking over the code for the amazon plugin, I did find one place where the plugin was making a handful of simultaneous requests to amazon, maybe that was trigerring the bot detection. I've inserted some sleeps -- this will make downloading slower, but hopefully more reliable.
|
![]() |
![]() |
![]() |
#55 |
Member
![]() Posts: 11
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
|
I just want to thank you for all of your massive work on this wonderful program.
So, the current plugin hits the site multiple times, much like the way windows hammers network shares? |
![]() |
![]() |
![]() |
#56 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,230
Karma: 23446430
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
No not really, the plugin actually hits the site less times than a typical browser request (browsers typically make between 5-10 simulataneous requests for a single domain) I've just further reduced to no more than one request a second.
|
![]() |
![]() |
![]() |
#57 |
Member
![]() Posts: 11
Karma: 10
Join Date: Nov 2016
Device: Kindle Desktop
|
This might sound like a moronic question but, what is the url to click on the CAPTCHA? i have looked but cant find it and amazons error is just as helpful as ANY from Microsoft.
|
![]() |
![]() |
![]() |
#58 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,230
Karma: 23446430
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
There is no URL basically accessing any page will redirect to the captcha. But it may or may nt happen with your regular browser since that will have a different set of cookies/profile.
|
![]() |
![]() |
![]() |
#59 |
Connoisseur
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 78
Karma: 457752
Join Date: Oct 2011
Location: Deep South
Device: 4 or 5 generations of kindles
|
So I've mentioned on another thread having this problem on and off for the last few months. In the last few weeks amazon has become unusable for me to download metadata. I have changed ip's (even what country my ip is in), what amazon site it gets metadata from, all kinds of things I have tried. once in a while it will give me 3 or 4 books but most often just days of doesn't work. I had gotten to copying from browser to comments field and typing in tags and series information by hand
![]() On reading this thread I saw the mention of the goodreads plugin and got it and started using it. Since it is owned by amazon I'm thinking all the KU books and self published will show up there also. I just did a search for 35 books and got back 27 results. So it is working better than amazon for me at the moment. I also want to add my thanks for calibre.. I have kicked in money a few times and will again. I use it a lot. I suggest for all of you that really do use it to kick back on paypal or other way to kovid a few dollars. I have over the years sent 10 to 25 dollars each time. I have some comercial software I have to use and have payed hundreds of dollars for. It doesn't work as well as this, doesn't get updated as often and bugs fixed as quickly. It really is worth our time to let kovid know how much we appreciate it. I think I will go do that right now ![]() Last edited by audeojude; 03-02-2017 at 10:03 AM. |
![]() |
![]() |
![]() |
#60 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,230
Karma: 23446430
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
I'm actualy currently working on modifying the amazon plugin to get the results from various search engine web caches, bing, google, wayback machine as well as amazon itself. Probably make it configurable so if one source stops working you can switch to another. That combined with the changes I made to further reduce query frequency should hopefully get it working again. If it doesn't then I'm afraid I am truly out of ideas.
|
![]() |
![]() |
![]() |
Thread Tools | Search this Thread |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Aussies launch anti-CAPTCHA petition. Is it time to kill CAPTCHAs? | Alexander Turcic | Lounge | 30 | 08-16-2013 07:32 PM |
Bug in Kobo processing of epub files causing hang in "Processing content" | BensonBear | Kobo Reader | 21 | 12-21-2012 05:47 AM |
Get books bug for Amazon UK | rustleg | Library Management | 1 | 10-21-2012 01:25 PM |