title: Web scraping hidden API - request headers, cookie and XSFR-TOKEN handling
body: <p>I found a json file that contains all the data of the page. I notice in browser that after downloading the webpage the js sends a POST request to load the json file. If I put the json file link directly in the browser it is a GET request. Is it a better practice to get the webpage and use the relevant cookies to request the json file instead of requesting the json link directly? This way looks more like a normal user?</p>
  <p>I also see in the POST and GET requests a XSRF-TOKEN was sent. I suppose this is needed in the request?</p>
  <p>Here are POST vs GET requests (stripped out the analytics and 3rd tool cookies). Which items in the headers are best to be included in the requests? Plan to rotate user-agent and IP.</p>
  <p>POST
  <br />
  Host:
  <br />
  Connection: keep-alive
  <br />
  Content-Length: 2
  <br />
  Accept: application/json, text/plain, <em>/</em>
  <br />
  X-XSRF-TOKEN:
  <br />
  ADRUM: isAjax:true
  <br />
  User-Agent:
  <br />
  Content-Type: application/json;charset=UTF-8
  <br />
  Origin:
  <br />
  Sec-Fetch-Site: same-origin
  <br />
  Sec-Fetch-Mode: cors
  <br />
  Sec-Fetch-Dest: empty
  <br />
  Referer:
  <br />
  Accept-Encoding: gzip, deflate, br
  <br />
  Accept-Language: en,en-US
  <br />
  Cookie: JSESSIONID=; XSRF-TOKEN=;</p>
  <p>GET
  <br />
  Host:
  <br />
  Connection:
  <br />
  Upgrade-Insecure-Requests: 1
  <br />
  User-Agent:
  <br />
  Accept:
  text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,<em>/</em>;q=0.8,application/signed-exchange;v=b3;q=0.9
  <br />
  Sec-Fetch-Site: none
  <br />
  Sec-Fetch-Mode: navigate
  <br />
  Sec-Fetch-Dest: document
  <br />
  Accept-Encoding: gzip, deflate, br
  <br />
  Accept-Language: en,en-US
  <br />
  Cookie: JSESSIONI=; XSRF-TOKEN=;</p>