Cloudfare problem

[thiago.eec]

Hi everyone.

I have a plugin called Skoob Sync. It uses webscraping to sync the user info. Now, the site started using Cloudfare when you try to login, giving me a 'HTTP Error 403: Forbidden'. The metadata plugin (Skoob Books) is not affected as it does not need to login.

I know this is not a calibre question, but since many of the plugin developers use webscraping, I thought that I might get some help here. Is there any way to bypass the Cloudfare check?

This is the relevant code for login:

Spoiler:

            self.opener = six.moves.urllib.request.build_opener(six.moves.urllib.request.HTTPCookieProcessor(self.cj))

            # Install our opener (note that this changes the global opener to the one
            # we just made, but you can also just call opener.open() if you want)
            six.moves.urllib.request.install_opener(self.opener)

            # Authentication page
            authentication_url = 'https://api.skoob.com.br/login'

            # Credentials
            payload = {
                'data[Usuario][email]': self.prefs['user'],
                'data[Usuario][senha]': self.key_password,
            }

            # Use urllib to encode the payload
            data = six.moves.urllib.parse.urlencode(payload).encode()

            # Build our Request object (supplying 'data' makes it a POST)
            login_req = six.moves.urllib.request.Request(authentication_url, data, headers=random_ua())

            login = six.moves.urllib.request.urlopen(login_req)

Spoiler:

# Get a random user agent from calibre. This is used on Skoob access.
def random_ua():
    try:
        from calibre import random_user_agent
        try:
            hdr = {'User-Agent': random_user_agent(allow_ie=False)}
            return hdr
        except TypeError:
            hdr = {'User-Agent': random_user_agent()}
            return hdr
    except ImportError:
        hdr = {'User-Agent': 'Mozilla/5.0 (Windows NT .1; Win64; x64)'}
        return hdr

[JimmXinu]

I'll offer a few pointers, but I haven't researched it recently.

Cloudfare has more than one level of blocking it can be running at for a given site.

Some lower levels can be bypassed with cloudscraper.

FlareSolverr is a proxy that runs a headless browser to handle requests. As I understand it, this uses it's own proxy API, not a standard one. Also, this proxy works for web pages, but not images or binaries. I don't believe it works for the highest "under attack" levels of Cloudfare.

FanFicFare has code that can use either of these, as well as code that can read cached pages out of your regular browser's cache directory. This is a pain, because you have to load the page in your browser first to cache it--and not all pages are cached.

The actual cache reading code isn't mine and I don't pretend to understand it more than superficially.

[kovidgoyal]

@JimmXinu: Just FYI as of calibre 7.17 there is a headless browser (chromium, via Qt WebEngine) in calibre you can use to get URLs. See the WebEngineBrowser class in scraper/qt.py

[thiago.eec]

Originally Posted by JimmXinu

I'll offer a few pointers, but I haven't researched it recently.

Cloudfare has more than one level of blocking it can be running at for a given site.

Some lower levels can be bypassed with cloudscraper.

FlareSolverr is a proxy that runs a headless browser to handle requests. As I understand it, this uses it's own proxy API, not a standard one. Also, this proxy works for web pages, but not images or binaries. I don't believe it works for the highest "under attack" levels of Cloudfare.

FanFicFare has code that can use either of these, as well as code that can read cached pages out of your regular browser's cache directory. This is a pain, because you have to load the page in your browser first to cache it--and not all pages are cached.

The actual cache reading code isn't mine and I don't pretend to understand it more than superficially.

Thank you for your help. I did try clouscraper and FlareSolverr, but could not make it work.

Originally Posted by kovidgoyal

@JimmXinu: Just FYI as of calibre 7.17 there is a headless browser (chromium, via Qt WebEngine) in calibre you can use to get URLs. See the WebEngineBrowser class in scraper/qt.py

I tried using WebEngineBroswer to access the login page (protected by Cloudfare) but seems like JavaScript is not enable by default. The HTML returned has this message: 'Enable JavaScript and cookies to continue'.
How can I enable full JavaScript and Cookies support for a WebEngineBrowser instance?

This is necessary because the Cloudfare check basically gives the browser a few JS calculations to solve, before allowing the user to see the login page.

[kovidgoyal]

It doesnt support javascript, once you enable javascript detecting that the browser is not a real browser is pretty easy so it wont help you with cloudflare.

[thiago.eec]

Originally Posted by kovidgoyal

It doesnt support javascript, once you enable javascript detecting that the browser is not a real browser is pretty easy so it wont help you with cloudflare.

I see. I'm out of options, then.

[JSWolf]

Originally Posted by thiago.eec

I see. I'm out of options, then.

See my reply in the Skoob thread.

[gbm]

Why don't you use a little courtesy and post a link.

bernie

Originally Posted by JSWolf

See my reply in the Skoob thread.

[JSWolf]

Originally Posted by gbm

Why don't you use a little courtesy and post a link.

bernie

The plugin thread for Skoob in the calibre forum as this isn't the correct thread to reply in.

https://www.mobileread.com/forums/sh...d.php?t=320753