Possible bug causing Amazon captcha? - Page 4

Oddball · 02-26-2017, 06:12 PM

I hate to say this, but maybe its time to figure out a way to create a site that images amazon without the captcha for the exclusive use of bots to use. MAYBE it is possible to use Wayback machine for that. A friend of mine data mined a site and create a mirror image of the original site. would just require a large storage of at least a TB or more to mirror. Just a thought.

PeterT · 02-26-2017, 06:32 PM

I seriously doubt Amazon would allow that

Oddball · 02-26-2017, 07:32 PM

Quote:

Originally Posted by PeterT

I seriously doubt Amazon would allow that

well truthfully they are already and have been apart of Archive.org since 2000 at least.....it is the one way i find info on some eBook they have discontinued.

http://amazon.com

Saved 52,539 times between December 12, 1998 and February 26, 2017.

PeterT · 02-26-2017, 08:27 PM

Fine go and make a metadata plugin to access archive.org's copy of Amazon then

kovidgoyal · 02-26-2017, 10:10 PM

It's not quite as simple as using a mirror. The problem is metadata plugins require a way to search for title/author/isbn/etc. So in addition to a mirror you'd need to implement search as well.

So directly using something like archive.org is not possible. What one can do is probably use google's sitesearch for archive.org. Though IIRC gogole does not like bots using their search either, so you'd likely run into the same problems as with amazon, eventually.

jhowell · 02-28-2017, 12:19 PM

In my personal experience it seems that Amazon doesn't bother users again for quite a while once they answer a captcha, as long as cookies are maintained.

Perhaps a solution would be to present the captcha to the user for solution, proving the non-botness of the requester along with preserving the Amazon cookies in a more permanent fashion.

ETA: The function of making a request to Amazon and having the user solve the captcha could be a UI plugin, separate from the Amazon metadata plugin, as long as they share cookies.

kovidgoyal · 02-28-2017, 12:44 PM

That's awfully fiddly, and probably if amazon detects that bots resume operations after answering a captcha on the same IP it will escalate to banning the IP with HTTP 503 or the like. After all, it is actually fairly trivial to solve captchas programmatically, the problem with doing that is that the next step is getting the IP banned, at least that is how I recall most bot detection algorithms work. A bit of googling suggests that is indeed how they work.

jhowell · 02-28-2017, 01:32 PM

I definitely do not want my IP address banned by Amazon. It is too bad that they are taking these steps to block programmatic access to their site.

kovidgoyal · 02-28-2017, 10:43 PM

Looking over the code for the amazon plugin, I did find one place where the plugin was making a handful of simultaneous requests to amazon, maybe that was trigerring the bot detection. I've inserted some sleeps -- this will make downloading slower, but hopefully more reliable.

Oddball · 02-28-2017, 11:09 PM

I just want to thank you for all of your massive work on this wonderful program.

So, the current plugin hits the site multiple times, much like the way windows hammers network shares?

kovidgoyal · 02-28-2017, 11:25 PM

No not really, the plugin actually hits the site less times than a typical browser request (browsers typically make between 5-10 simulataneous requests for a single domain) I've just further reduced to no more than one request a second.

Oddball · 03-01-2017, 09:05 AM

This might sound like a moronic question but, what is the url to click on the CAPTCHA? i have looked but cant find it and amazons error is just as helpful as ANY from Microsoft.

kovidgoyal · 03-01-2017, 09:15 AM

There is no URL basically accessing any page will redirect to the captcha. But it may or may nt happen with your regular browser since that will have a different set of cookies/profile.

audeojude · 03-02-2017, 09:57 AM

So I've mentioned on another thread having this problem on and off for the last few months. In the last few weeks amazon has become unusable for me to download metadata. I have changed ip's (even what country my ip is in), what amazon site it gets metadata from, all kinds of things I have tried. once in a while it will give me 3 or 4 books but most often just days of doesn't work. I had gotten to copying from browser to comments field and typing in tags and series information by hand

On reading this thread I saw the mention of the goodreads plugin and got it and started using it. Since it is owned by amazon I'm thinking all the KU books and self published will show up there also. I just did a search for 35 books and got back 27 results. So it is working better than amazon for me at the moment.

I also want to add my thanks for calibre.. I have kicked in money a few times and will again. I use it a lot. I suggest for all of you that really do use it to kick back on paypal or other way to kovid a few dollars. I have over the years sent 10 to 25 dollars each time. I have some comercial software I have to use and have payed hundreds of dollars for. It doesn't work as well as this, doesn't get updated as often and bugs fixed as quickly. It really is worth our time to let kovid know how much we appreciate it. I think I will go do that right now

Words are nice but actions speak greatly. I maintain a community website and when members send me a few dollars in thanks for the time and effort I put in it really means a lot to me.

kovidgoyal · 03-02-2017, 10:03 AM

I'm actualy currently working on modifying the amazon plugin to get the results from various search engine web caches, bing, google, wayback machine as well as amazon itself. Probably make it configurable so if one source stops working you can switch to another. That combined with the changes I made to further reduce query frequency should hopefully get it working again. If it doesn't then I'm afraid I am truly out of ideas.

02-26-2017, 06:12 PM	#46
Oddball Member Posts: 11 Karma: 10 Join Date: Nov 2016 Device: Kindle Desktop	I hate to say this, but maybe its time to figure out a way to create a site that images amazon without the captcha for the exclusive use of bots to use. MAYBE it is possible to use Wayback machine for that. A friend of mine data mined a site and create a mirror image of the original site. would just require a large storage of at least a TB or more to mirror. Just a thought. Last edited by Oddball; 02-26-2017 at 06:16 PM.

02-28-2017, 12:19 PM	#51
jhowell Grand Sorcerer Posts: 6,610 Karma: 84812983 Join Date: Nov 2011 Location: Tampa Bay, Florida Device: Kindles	In my personal experience it seems that Amazon doesn't bother users again for quite a while once they answer a captcha, as long as cookies are maintained. Perhaps a solution would be to present the captcha to the user for solution, proving the non-botness of the requester along with preserving the Amazon cookies in a more permanent fashion. ETA: The function of making a request to Amazon and having the user solve the captcha could be a UI plugin, separate from the Amazon metadata plugin, as long as they share cookies. Last edited by jhowell; 02-28-2017 at 12:25 PM.

03-02-2017, 09:57 AM	#59
audeojude Connoisseur Posts: 66 Karma: 14170 Join Date: Oct 2011 Device: kindle 1	So I've mentioned on another thread having this problem on and off for the last few months. In the last few weeks amazon has become unusable for me to download metadata. I have changed ip's (even what country my ip is in), what amazon site it gets metadata from, all kinds of things I have tried. once in a while it will give me 3 or 4 books but most often just days of doesn't work. I had gotten to copying from browser to comments field and typing in tags and series information by hand On reading this thread I saw the mention of the goodreads plugin and got it and started using it. Since it is owned by amazon I'm thinking all the KU books and self published will show up there also. I just did a search for 35 books and got back 27 results. So it is working better than amazon for me at the moment. I also want to add my thanks for calibre.. I have kicked in money a few times and will again. I use it a lot. I suggest for all of you that really do use it to kick back on paypal or other way to kovid a few dollars. I have over the years sent 10 to 25 dollars each time. I have some comercial software I have to use and have payed hundreds of dollars for. It doesn't work as well as this, doesn't get updated as often and bugs fixed as quickly. It really is worth our time to let kovid know how much we appreciate it. I think I will go do that right now Words are nice but actions speak greatly. I maintain a community website and when members send me a few dollars in thanks for the time and effort I put in it really means a lot to me. Last edited by audeojude; 03-02-2017 at 10:03 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Aussies launch anti-CAPTCHA petition. Is it time to kill CAPTCHAs?	Alexander Turcic	Lounge	30	08-16-2013 07:32 PM
Bug in Kobo processing of epub files causing hang in "Processing content"	BensonBear	Kobo Reader	21	12-21-2012 05:47 AM
Get books bug for Amazon UK	rustleg	Library Management	1	10-21-2012 01:25 PM

02-26-2017, 06:32 PM	#47
PeterT Grand Sorcerer Posts: 12,395 Karma: 74317822 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	I seriously doubt Amazon would allow that

02-26-2017, 08:27 PM	#49
PeterT Grand Sorcerer Posts: 12,395 Karma: 74317822 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	Fine go and make a metadata plugin to access archive.org's copy of Amazon then

02-26-2017, 10:10 PM	#50
kovidgoyal creator of calibre Posts: 44,117 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's not quite as simple as using a mirror. The problem is metadata plugins require a way to search for title/author/isbn/etc. So in addition to a mirror you'd need to implement search as well. So directly using something like archive.org is not possible. What one can do is probably use google's sitesearch for archive.org. Though IIRC gogole does not like bots using their search either, so you'd likely run into the same problems as with amazon, eventually.

02-28-2017, 12:44 PM	#52
kovidgoyal creator of calibre Posts: 44,117 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	That's awfully fiddly, and probably if amazon detects that bots resume operations after answering a captcha on the same IP it will escalate to banning the IP with HTTP 503 or the like. After all, it is actually fairly trivial to solve captchas programmatically, the problem with doing that is that the next step is getting the IP banned, at least that is how I recall most bot detection algorithms work. A bit of googling suggests that is indeed how they work.

02-28-2017, 01:32 PM	#53
jhowell Grand Sorcerer Posts: 6,610 Karma: 84812983 Join Date: Nov 2011 Location: Tampa Bay, Florida Device: Kindles	I definitely do not want my IP address banned by Amazon. It is too bad that they are taking these steps to block programmatic access to their site.

02-28-2017, 10:43 PM	#54
kovidgoyal creator of calibre Posts: 44,117 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Looking over the code for the amazon plugin, I did find one place where the plugin was making a handful of simultaneous requests to amazon, maybe that was trigerring the bot detection. I've inserted some sleeps -- this will make downloading slower, but hopefully more reliable.

02-28-2017, 11:09 PM	#55
Oddball Member Posts: 11 Karma: 10 Join Date: Nov 2016 Device: Kindle Desktop	I just want to thank you for all of your massive work on this wonderful program. So, the current plugin hits the site multiple times, much like the way windows hammers network shares?

02-28-2017, 11:25 PM	#56
kovidgoyal creator of calibre Posts: 44,117 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No not really, the plugin actually hits the site less times than a typical browser request (browsers typically make between 5-10 simulataneous requests for a single domain) I've just further reduced to no more than one request a second.

03-01-2017, 09:05 AM	#57
Oddball Member Posts: 11 Karma: 10 Join Date: Nov 2016 Device: Kindle Desktop	This might sound like a moronic question but, what is the url to click on the CAPTCHA? i have looked but cant find it and amazons error is just as helpful as ANY from Microsoft.

03-01-2017, 09:15 AM	#58
kovidgoyal creator of calibre Posts: 44,117 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There is no URL basically accessing any page will redirect to the captcha. But it may or may nt happen with your regular browser since that will have a different set of cookies/profile.

Advert

Advert

03-02-2017, 10:03 AM	#60
kovidgoyal creator of calibre Posts: 44,117 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I'm actualy currently working on modifying the amazon plugin to get the results from various search engine web caches, bing, google, wayback machine as well as amazon itself. Probably make it configurable so if one source stops working you can switch to another. That combined with the changes I made to further reduce query frequency should hopefully get it working again. If it doesn't then I'm afraid I am truly out of ideas.