title: Java jsoup html parsing robot index / bot detection, noindex
body: <p>Firstly, <strong>this is not a duplicated question</strong>, because I have already checked almost all 503 / robot index problems. None of them solved my problem. I am trying to get giveaway list from indiegala.com but this site has some kind of protection to prevent bots and robots. My purpose is not illegal, I just want to get giveaway list then check games whether they have steam trade cards or not. But right know, indiegala gives me a robot index. Currently I am using that code;</p>
  
  <pre><code>       String url = "https://www.indiegala.com/giveaways";
      try {
          String content = Jsoup.connect(url).userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36").ignoreHttpErrors(true).followRedirects(true).get().html();
          System.out.println(content);
      } catch (IOException ex) {
          System.out.println(ex.toString());
      }
  </code></pre>
  
  <p>To see the output(source of the site, in my code, variable "content"), you can run the code that I gave, I cannot add output here because it is a little bit long. But it looks like that;</p>
  
  <pre><code>&lt;head&gt;
   &lt;meta name="ROBOTS" content="NOINDEX, NOFOLLOW" /&gt;
  &lt;/head&gt;
  </code></pre>
  
  <p>So how can I pass this protection ? Can my program pretend like a human to pass this protection ?</p>