Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Request is being blocked. #32

Open
muzaT opened this issue Sep 18, 2024 · 0 comments
Open

API Request is being blocked. #32

muzaT opened this issue Sep 18, 2024 · 0 comments

Comments

@muzaT
Copy link

muzaT commented Sep 18, 2024

I recently came across this python package and really appreciate the developers that they have put soo much effort and made it work flawless.

I tried to access a website, after inserting headers as they go in the browser. Web page content is fetched successfully but the issue is that, there is a button. Clicking on that button, triggers an API request, which returns json data. When I try it with this package, it gets blocked and returns below error:

<html>
    <head>
        <noscript>
            <title>Pardon Our Interruption</title>
        </noscript>

        <meta name="viewport" content="width=1000">
        <meta name="robots" content="noindex, nofollow">
        <meta http-equiv="cache-control" content="no-cache, no-store, must-revalidate">
        <meta http-equiv="pragma" content="no-cache">
        <meta http-equiv="expires" content="0">

        <style>
            .container { max-width: 800px; margin: auto; font-family: 'Helvetica Neue',Helvetica,Arial,sans-serif; color: #7a838c; }
            h1 { color: #2a2d30; font-weight: 500; }
            li { margin: 0 0 10px; }
            a { color: #428bca; }
            a:hover, a:focus { color: #2a6496; }
        </style>

        <script>
          var isSpa = new URLSearchParams(window.location.search).get('X-SPA') === '1' || window.isImpervaSpaSupport;
        </script>

        <!-- This head template should be placed before the following script tag that loads the challenge script -->
        <script>
          window.onProtectionInitialized = function(protection) {
            if (protection && protection.cookieIsSet && !protection.cookieIsSet()) {
              showBlockPage();
              return;
            }
            if (!isSpa) {
              window.location.reload(true);
            }
          };
          window.reeseSkipExpirationCheck = true;
        </script>

        <script>
          if (!isSpa) {
            var scriptElement = document.createElement('script');
            scriptElement.type = "text/javascript";
            scriptElement.src = "/Spurre-Onell-vp-Enter-feed-ere-Yourthe-away-riso/14883154119434262891?s=9lRPm776";
            scriptElement.async = true;
            scriptElement.defer = true;
            document.head.appendChild(scriptElement);
          }
        </script>
        
    </head>
    <body>

        

        <div class="container">
            <script>document.getElementsByClassName("container")[0].style.display = "none";</script>
            
            <h1>Pardon Our Interruption</h1>
<p>As you were browsing something about your browser made us think you were a bot. There are a few reasons this might happen:</p>
<ul>
<noscript><li>You've disabled JavaScript in your web browser.</li></noscript>
<li>You're a power user moving through this website with super-human speed.</li>
<li>You've disabled cookies in your web browser.</li>
<li>A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this <a title='Third party browser plugins that block javascript' href='http://ds.tl/help-third-party-plugins' target='_blank'>support article</a>.</li>
</ul>
<p>To regain access, please make sure that cookies and JavaScript are enabled before reloading the page.</p>


        </div>
        <script>
          function showBlockPage() {
            document.title = "Pardon Our Interruption";
            document.getElementsByClassName("container")[0].style.display = "block";
          }

          if (isSpa) {
            showBlockPage();
          } else {
            setTimeout(showBlockPage, 10000);
          }
        </script>
    </body>
</html>


Process finished with exit code 0

Code snippet:

session = IncapSession()

session.cookies.set('Cookie', 'visid_incap_2413658=OtjS12IzRQ+hKNsLDp4Shx2Q6mYAAAAAQUIPAAAAAADzUMUxSF+SiN0hHLBWY6rT; incap_ses_128_2413658=rkSPJMtdQ1YZXSCscL/GAR2Q6mYAAAAA0vLjT1iXPBwUoEM1S3dxqQ==; incap_ses_775_2413658=q1Rtb5x0Rhk1Ol4bg1rBCh+Q6mYAAAAAFW5niDtjVSQ0uF1tQS4Pqg==; _gcl_au=1.1.512602668.1726648355; _tt_enable_cookie=1; _ttp=G3ss00DM3AsDi8LjliofUogZ97b; _ga=GA1.1.771662499.1726648356; _scid=iIkg4MW064d6cqdRAxN8Wree7GrWNTlP; _fbp=fb.1.1726648355756.490626892793758522; _ScCbts=%5B%5D; _scid_r=gQkg4MW064d6cqdRAxN8Wree7GrWNTlPzguJmQ; nlbi_2413658_2147483392=tIHqG/lUvTLaYOb6eguELgAAAAC3J3vBZqQUgvfc3qBK4kQa; reese84=3:HcPPj2rdqpDwoWlYUaCduQ==:9wy/KN76yAp+IX321b5vQXWWhI6SR8gKljxSCdaDzDdO5+5tLigqXWyJ6HZSt9hjkjrChBgZWaWM8Up2yOM/6Bch9xTuQrMwF/Cy5MnP3rnL7ObFuHmpquPM0TAUjikDuFG22fDAAMkP6voelGgbwVRT0Zg8E36Dz4f7fIdCrd80wq0Dm3b9qTAKSzkOSERBDlCdz2deSP/jzQyn6kZb6xwvb/6MS6SD+dPw1SL84xOtNHuIF74vXvD7wSvM/xx2oXOkrFur+N1wCtq9Dbvox0VgoVudpwY+6138vbzPlijbdPs1THZbAaw7vVA7Xwfeklhqvl7EIeFvmZFCJTpz4jffQZh/S0NEuZmxo41LCmb/USZH+62+W4+zKAT9E7r2Wsf5W6GdzwtVY8hl/Or8VwtMcyy2RECnOYWOUr2VuP1OYQQwToDHlEx8ofgRbROAHE3WZuKhkb37Ov7nody6Kw==:CJsPUKj0+ufJL0hAvmlMwiPWLoKge3DS+z4JBO9YRIc=; ias=0; nlbi_2413658=PuJtcKXdChm2y1yleguELgAAAAB3AhLXlU/LPxXcc8cn127l; _sctr=1%7C1726603200000; __gads=ID=ee5a178be1b0f05b:T=1726648356:RT=1726648356:S=ALNI_MZyw5qAhL0xPStB1n0AO9wKPZbejw; __gpi=UID=00000f0545f704b8:T=1726648356:RT=1726648356:S=ALNI_MYrEpGfB3M0ndF1X9ENeU-rzw_IGA; __eoi=ID=9ff938aca5b93c6d:T=1726648356:RT=1726648356:S=AA-AfjYiQlITvzkWMzSsBbMpghn4; _cc_id=d385bcb2a3e0f8df6d9d5690b0cc21d0; panoramaId_expiry=1726734756911; panoramaId=e231123d24848fca3d292205f7dda9fb927a63c68d47f55451f18511fdcfb078; panoramaIdType=panoDevice; moe_uuid=39a77911-d741-4354-b8e0-5a4645d10df1; USER_DATA=%7B%22attributes%22%3A%5B%5D%2C%22subscribedToOldSdk%22%3Afalse%2C%22deviceUuid%22%3A%2239a77911-d741-4354-b8e0-5a4645d10df1%22%2C%22deviceAdded%22%3Atrue%7D; _gaexp=GAX1.2.xB4nT9jdK5vQz7UwLpR3yH.20074.0; OPT_IN_SHOWN_TIME=1726648362280; __rtbh.uid=%7B%22eventType%22%3A%22uid%22%2C%22id%22%3A%22771662499.1726648356%22%7D; __rtbh.lid=%7B%22eventType%22%3A%22lid%22%2C%22id%22%3A%22kAppdxPpSxBnMaq25HOn%22%7D; SOFT_ASK_STATUS=%7B%22actualValue%22%3A%22dismissed%22%2C%22MOE_DATA_TYPE%22%3A%22string%22%7D; lotame_domain_check=dubizzle.com; _ga_LRML1YM9GH=GS1.1.1726648355.1.1.1726648387.28.0.0; SESSION=%7B%22sessionKey%22%3A%2273eea8e6-6c92-480e-8a23-047177ef6aa4%22%2C%22sessionStartTime%22%3A%222024-09-18T08%3A32%3A40.070Z%22%2C%22sessionMaxTime%22%3A1800%2C%22customIdentifiersToTrack%22%3A%5B%5D%2C%22sessionExpiryTime%22%3A1726650187904%2C%22numberOfSessions%22%3A1%7D; UniqueLead=$1237-15945764')
response = session.get('https://dubai.dubizzle.com/api/v4/leads/1237/0a7018d4f4cc46428e05e644e8ded161/listing-profile/phone/', headers={
        "Accept": "application/json",
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9',
        'content-type': 'application/json;charset=UTF-8',
        'cookie': 'visid_incap_2413658=OtjS12IzRQ+hKNsLDp4Shx2Q6mYAAAAAQUIPAAAAAADzUMUxSF+SiN0hHLBWY6rT; incap_ses_128_2413658=rkSPJMtdQ1YZXSCscL/GAR2Q6mYAAAAA0vLjT1iXPBwUoEM1S3dxqQ==; incap_ses_775_2413658=q1Rtb5x0Rhk1Ol4bg1rBCh+Q6mYAAAAAFW5niDtjVSQ0uF1tQS4Pqg==; _gcl_au=1.1.512602668.1726648355; _tt_enable_cookie=1; _ttp=G3ss00DM3AsDi8LjliofUogZ97b; _ga=GA1.1.771662499.1726648356; _scid=iIkg4MW064d6cqdRAxN8Wree7GrWNTlP; _fbp=fb.1.1726648355756.490626892793758522; _ScCbts=%5B%5D; reese84=3:HcPPj2rdqpDwoWlYUaCduQ==:9wy/KN76yAp+IX321b5vQXWWhI6SR8gKljxSCdaDzDdO5+5tLigqXWyJ6HZSt9hjkjrChBgZWaWM8Up2yOM/6Bch9xTuQrMwF/Cy5MnP3rnL7ObFuHmpquPM0TAUjikDuFG22fDAAMkP6voelGgbwVRT0Zg8E36Dz4f7fIdCrd80wq0Dm3b9qTAKSzkOSERBDlCdz2deSP/jzQyn6kZb6xwvb/6MS6SD+dPw1SL84xOtNHuIF74vXvD7wSvM/xx2oXOkrFur+N1wCtq9Dbvox0VgoVudpwY+6138vbzPlijbdPs1THZbAaw7vVA7Xwfeklhqvl7EIeFvmZFCJTpz4jffQZh/S0NEuZmxo41LCmb/USZH+62+W4+zKAT9E7r2Wsf5W6GdzwtVY8hl/Or8VwtMcyy2RECnOYWOUr2VuP1OYQQwToDHlEx8ofgRbROAHE3WZuKhkb37Ov7nody6Kw==:CJsPUKj0+ufJL0hAvmlMwiPWLoKge3DS+z4JBO9YRIc=; ias=0; nlbi_2413658=PuJtcKXdChm2y1yleguELgAAAAB3AhLXlU/LPxXcc8cn127l; _sctr=1%7C1726603200000; _cc_id=d385bcb2a3e0f8df6d9d5690b0cc21d0; panoramaId_expiry=1726734756911; panoramaId=e231123d24848fca3d292205f7dda9fb927a63c68d47f55451f18511fdcfb078; panoramaIdType=panoDevice; moe_uuid=39a77911-d741-4354-b8e0-5a4645d10df1; USER_DATA=%7B%22attributes%22%3A%5B%5D%2C%22subscribedToOldSdk%22%3Afalse%2C%22deviceUuid%22%3A%2239a77911-d741-4354-b8e0-5a4645d10df1%22%2C%22deviceAdded%22%3Atrue%7D; _gaexp=GAX1.2.xB4nT9jdK5vQz7UwLpR3yH.20074.0; OPT_IN_SHOWN_TIME=1726648362280; __rtbh.uid=%7B%22eventType%22%3A%22uid%22%2C%22id%22%3A%22771662499.1726648356%22%7D; __rtbh.lid=%7B%22eventType%22%3A%22lid%22%2C%22id%22%3A%22kAppdxPpSxBnMaq25HOn%22%7D; SOFT_ASK_STATUS=%7B%22actualValue%22%3A%22dismissed%22%2C%22MOE_DATA_TYPE%22%3A%22string%22%7D; UniqueLead=$1237-15945764; skybar_sess_False=1; default_site=2; sid=mhh3b6u0dhf44r7q2oji1cvvou9a7f19; nlbi_2413658_2147483392=+JAvJYmuMznMnfcreguELgAAAADPTu7sOTLSpa1e1C6Y4+6u; _scid_r=kgkg4MW064d6cqdRAxN8Wree7GrWNTlPzguJmg; __gads=ID=ee5a178be1b0f05b:T=1726648356:RT=1726649146:S=ALNI_MZyw5qAhL0xPStB1n0AO9wKPZbejw; __gpi=UID=00000f0545f704b8:T=1726648356:RT=1726649146:S=ALNI_MYrEpGfB3M0ndF1X9ENeU-rzw_IGA; __eoi=ID=9ff938aca5b93c6d:T=1726648356:RT=1726649146:S=AA-AfjYiQlITvzkWMzSsBbMpghn4; lotame_domain_check=dubizzle.com; _ga_LRML1YM9GH=GS1.1.1726648355.1.1.1726649161.42.0.0; SESSION=%7B%22sessionKey%22%3A%2273eea8e6-6c92-480e-8a23-047177ef6aa4%22%2C%22sessionStartTime%22%3A%222024-09-18T08%3A32%3A40.070Z%22%2C%22sessionMaxTime%22%3A1800%2C%22customIdentifiersToTrack%22%3A%5B%5D%2C%22sessionExpiryTime%22%3A1726650961599%2C%22numberOfSessions%22%3A1%7D',
        'priority': 'u=1, i',
        'referer': 'https://dubai.dubizzle.com/motors/used-cars/chevrolet/camaro/2024/5/27/ss-fifty-50th-anniversary-first-owner-full-2-973---0a7018d4f4cc46428e05e644e8ded161/',
        'sec-ch-ua': '"Chromium";v="128", "Not;A=Brand";v="24", "Google Chrome";v="128"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-origin',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36',
        'x-access-token': 'eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6ImUwOGEyOWFiLWI4YmUtNGE5Ni04M2M0LTExYmM3ZjIwZTdkNyIsImlzcyI6ImR1Yml6emxlLmNvbSIsImF1ZCI6WyJkdWJpenpsZS5jb20iXSwic3ViIjoiZTA4YTI5YWItYjhiZS00YTk2LTgzYzQtMTFiYzdmMjBlN2Q3IiwiZXhwIjoxNzI2NzM1NTQxLCJpYXQiOjE3MjY2NDkxNDEsImp0aSI6IjBlYjdjZjQyLTY0MGYtNGVkOS1iOTMxLTM1ZjQyYjFjYzY1NSIsInR5cCI6IkFjY2VzcyBKV1QgVG9rZW4iLCJmbGFncyI6eyJsb2dnZWRfaW4iOmZhbHNlLCJpc19zdGFmZiI6ZmFsc2UsImlzX3N1cGVydXNlciI6ZmFsc2UsImlzX3Byb3BlcnR5X2FnZW50IjpmYWxzZSwiaXNfbGFuZGxvcmQiOmZhbHNlLCJpc19tb3RvcnNfYWdlbnQiOmZhbHNlLCJpc19qb2JzX2FnZW50IjpmYWxzZSwiaXNfbGVhZF9ibG9jayI6ZmFsc2UsImlzX2xlYWRfc29mdF9ibG9jayI6ZmFsc2UsImlzX2NoYXRfYmxvY2siOmZhbHNlLCJpc19jaGF0X3NvZnRfYmxvY2siOmZhbHNlLCJoYXNfY2FsbF90cmFja2luZyI6ZmFsc2UsImNhbl9yZXBvcnQiOnRydWUsImhpZGVfcHVibGljX3Byb2ZpbGUiOmZhbHNlLCJpc19tb3RvcnNfYWdlbnRfZW1wbG95ZWVfdXNlciI6ZmFsc2UsImlzX3JlbnRhbF9jYXJzX2FnZW50IjpmYWxzZX0sInVzZXJfZGF0YSI6eyJ1c2VyX2lkIjpudWxsLCJnZW5kZXIiOm51bGwsIm5hdGlvbmFsaXR5IjpudWxsLCJlZHVjYXRpb24iOm51bGwsInJvbGUiOm51bGwsImRvYiI6bnVsbCwiYWdlIjpudWxsLCJmaXJzdF9uYW1lIjoiIiwibGFzdF9uYW1lIjoiIiwiZW1haWwiOiIiLCJwaG9uZSI6IiIsInZlcmlmaWNhdGlvbl9zdGF0dXMiOiJub3RfYXBwbGllZCIsInZlcmlmaWNhdGlvbl9zdGFydF9kYXRlIjpudWxsfX0.J9UjbnprI-uyW6XSefhjS_sSebSFZcRJl2wK3mMrsGvxKA_by5EnSMpN5TRS10_5ZSDUe3vMm9CtnjQo7gfP5NWfKFbZGCzIAA987rKXEbcvNiZ6077obZR0kpqDXGE5gp3fx3v-2KOy0XKzRU4Aaag8gMmUkRPWJ2hhdsJJlY1cZqCVGuWbefz_YK9IqN77vEC-qoKdW9cgNtWlzW9PdbO39C6-79BMSa9or0Uj0LfZdtC5-I3ZpCZXb7XmOuv2WJ9w_orN5eHebDvoLCouHjQCT0GKj0XovLUpFJI60lU9HmloMktBk-iqqtc2NsPpc4Fda7sw_pT_QmY13MasOIlJFVpn0oa_in5g2xPN-IVdXstc6n42SOG39egNxzp2nR1XNQo9kaJ_kSPgqbtN7V2ADvzeIGdExULtVOTBEoWrrv1-ybSB681xYtcrJfVUkoccRKSw6gslBU4oKps0oRTf1pXb8C5ePN8AtUG01EJW9qq9zqNoCg7_3-2Tr9Z1dmMyy9wpy4kq8fLhyUvWIlwjPcskAVJhx7vkkjuWtlQTPEcyPlRW6TPVOmZ4Z9g0tnZc8YYKPo29c5NK8S9AW9fYvnOUm1EvXjKDGrBanbujX_niGgX5HzAsqMFwOIB2z_Rbvw-hLdZEu5_ZiPd5oUtutOEUwz9Zob60Me5ZOzc',
        # Add other headers as needed
    })

print(response.text)

URL Example:
https://dubai.dubizzle.com/motors/used-cars/chevrolet/camaro/2024/5/27/ss-fifty-50th-anniversary-first-owner-full-2-973---0a7018d4f4cc46428e05e644e8ded161/

Click Show Phone Number button.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant