title: Captcha using requests even after changing headers and IP. How am I being tracked?
body: <p>I am trying to scrape some articles from xyz However, after a certain number of scrapes, a captcha appears.</p>
  <p>However, I am running into major issues.</p>
  <ol>
  <li><p>I am using <code>from fake_useragent import UserAgent</code> to randomize my header.</p>
  </li>
  <li><p>I am using random sleep times between requests</p>
  </li>
  <li><p>I am changing IP address using a VPN once a captcha appears. However, somehow a captcha still appears once my IP address appears.</p>
  </li>
  </ol>
  <p>It is also strange because while a captcha appears in the request response, a captcha does not appear in the browser.</p>
  <p>So, I assume that by header is just wrong.</p>
  <p>I turned off js and cookies when obtaining this request because with cookie and js, there is clearly info that the website is tracking me with.</p>
  <pre><code>headers = {
      &quot;authority&quot;: &quot;seekingalpha.com&quot;,
      &quot;method&quot;: &quot;GET&quot;,
      &quot;path&quot;: &quot;/article/4230872-dillards-still-room-downside&quot;,
      &quot;scheme&quot;: &quot;https&quot;,
      &quot;accept&quot;: &quot;text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8&quot;,
      &quot;accept-encoding&quot;: &quot;gzip, deflate, br&quot;,
      &quot;accept-language&quot;: 'en-US,en;q=0.9',
      &quot;upgrade-insecure-requests&quot;: &quot;1&quot;,
      &quot;user-agent&quot;: RANDOM
  }
  </code></pre>
  <p>This is close to what the website uses: They add</p>
  <pre><code>&quot;cache-control&quot;: &quot;max-age=0&quot;,
  &quot;if-none-match&quot;: 'W/&quot;6f11a6f9219176fda72f3cf44b0a2059&quot;',
  </code></pre>
  <p>This to my research is etags which is used for carching and can be use to track people. The <code>'W/...'</code> changes each request.</p>
  <p>Also, when I use <a href="https://github.com/JazzCore/python-pdfkit" rel="nofollow noreferrer">wkhtmltopdf</a> to print the screen as pdf, I a captcha never appears. I have also tried using selenium which is even worse. In addition, I have tried using proxies as seen <a href="https://stackoverflow.com/questions/48756326/web-scraping-results-in-403-forbidden-error">here</a>.</p>
  <p>So there definitely is a way of doing this. However, I am not doing it correctly. Does anyone have an idea what I am doing wrong?</p>
  <p>Edit:</p>
  <ol>
  <li><p>Sessions does not seems to be working</p>
  </li>
  <li><p>Random headers does not seem to be working</p>
  </li>
  <li><p>Random sleeps does not seem to be working</p>
  </li>
  <li><p>I am able to access the webpage using my VPN. Even once a capcha appears using requests, there is no captcha on the website in the browser.</p>
  </li>
  <li><p>Selenium does not work.</p>
  </li>
  <li><p>I really do not want to pay for a service to solve capchas.</p>
  </li>
  </ol>
  <p>I believe the issue is that I am not mimicking the browser well enough.</p>