title: third party website scraping without API
body: <p>I need to scrap <a href="http://www.thessaloniki.gr/egov/budget.html" rel="nofollow noreferrer">this</a> website, according to the choices made on a couple of combo boxes.
  The problem is that there is no API and the URL is constant so I can not copy the url to match the criteria.</p>
  
  <p>I managed, using the firefox inspector, to figure out the post command,
  <img src="https://i.stack.imgur.com/9XyKi.png" alt="enter image description here">
   but when using it in a java program as such:</p>
  
  <pre><code>String url = "  https://gaiacrmkea.c-gaia.gr/city_thessaloniki/index.php";
      URL obj = new URL(url);
      HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
  
      //add reuqest header
      con.setRequestMethod("POST");
      con.setRequestProperty("Accept-Language", "el-GR,el;q=0.8,en-US;q=0.5,en;q=0.3");
      String urlParameters = "fyear=2015&amp;esex=0&amp;cdief=Όλες";
      // Send post request
      con.setDoOutput(true);
      DataOutputStream wr = new DataOutputStream(con.getOutputStream());
      wr.writeBytes(urlParameters);
      wr.flush();
      wr.close();
  
      int responseCode = con.getResponseCode();
      System.out.println("\nSending 'POST' request to URL : " + url);
      System.out.println("Post parameters : " + urlParameters);
      System.out.println("Response Code : " + responseCode);
  
      BufferedReader in = new BufferedReader(
              new InputStreamReader(con.getInputStream()));
      String inputLine;
      StringBuffer response = new StringBuffer();
  
      while ((inputLine = in.readLine()) != null) {
          response.append(inputLine+"\n");
  
      }
      in.close();
  
      //print result
      System.out.println(response.toString());
  </code></pre>
  
  <p>doesn't work. It only returns the graphics and the buttons, but no real data.
  When I copy the parameters and paste it to firefox, as such
  <a href="https://gaiacrmkea.c-gaia.gr/city_thessaloniki/index.php?fyear=2015&amp;esex=0&amp;cdief=%CE%8C%CE%BB%CE%B5%CF%82" rel="nofollow noreferrer">https://gaiacrmkea.c-gaia.gr/city_thessaloniki/index.php?fyear=2015&amp;esex=0&amp;cdief=Όλες</a></p>
  
  <p>I am getting it just right.</p>
  
  <p>any ideas?</p>
  
  <p>Edit:
  I would like that to be done automatically every day.</p>
  
  <p>Edit2:
  SOLVED!
  Based on the code provided by Özhan Düz I added these</p>
  
  <pre><code>Select type= document
                  .query("#esex")
                  .getSelect();
              type.setSelectedIndex(1);
              type.change();
  
              Select dep =document
                      .query("#cdief")
                      .getSelect();
  
              dep.setSelectedIndex(1);
              dep.change();
  </code></pre>
  
  <p>in order to get all three comboboxes.
  now i only need to do this automatically executed.</p>