title: Web scraping with jsoup and selenium
body: <p>I want to extract some information from this dynamic website with selenium and jsoup. To get the information I want to extract I have to click to the button &quot;Details öffnen&quot;. The first picture shows the website before cklicking the button and the second shows the website after clicking the button. The red marked information is the information I want to extract.</p>
  <p><img src="https://i.stack.imgur.com/IZmiK.png" alt="enter image description here" /></p>
  <p><img src="https://i.stack.imgur.com/7drq7.png" alt="enter image description here" /></p>
  <p>I first tried to extract the information only with Jsoup, but as I was told Jsoup can not handle dynamic content, so I am now trying to extract the Information with selenium and Jsoup like you can see in the sourcecode. Howerver I am not really sure if selenium is the right thing for this, so maybe there are other ways to extract the information I need more simple, but it is important that this could be done with Java.</p>
  <p>The next two pictures show the html code before clicking the button and after clicking the button.</p>
  <p><img src="https://i.stack.imgur.com/GW5e8.png" alt="enter image description here" /></p>
  <p><img src="https://i.stack.imgur.com/uZCcv.png" alt="enter image description here" /></p>
  <pre><code>public static void main(String[] args) {
      
      WebDriver driver = new FirefoxDriver(createFirefoxProfile());
      driver.get(&quot;http://www.seminarbewertung.de/seminar-bewertungen?id=3448&quot;);
      //driver.findElement(By.cssSelector(&quot;input[type='button'][value='Details öffnen']&quot;)).click();
      WebElement webElement = driver.findElement(By.cssSelector(&quot;input[type='submit'][value='Details öffnen'][rating_id='2318']&quot;));
      JavascriptExecutor executor = (JavascriptExecutor)driver;
      executor.executeScript(&quot;arguments[0].click();&quot;, webElement);
      String html_content = driver.getPageSource();
      //driver.close();
      
      
      Document doc1 = Jsoup.parse(html_content);
      System.out.println(&quot;Hallo&quot;);
      
      Elements elements = doc1.getAllElements();
      for (Element element : elements) {
          System.out.println(element);
      }
  
  }
  
  private static FirefoxProfile createFirefoxProfile() {
      File profileDir = new File(&quot;/tmp/firefox-profile-dir&quot;);
      if (profileDir.exists())
          return new FirefoxProfile(profileDir);
      FirefoxProfile firefoxProfile = new FirefoxProfile();
      File dir = firefoxProfile.layoutOnDisk();
      try {
          profileDir.mkdirs();
          FileUtils.copyDirectory(dir, profileDir);
      } catch (IOException e) {
          e.printStackTrace();
      }
      return firefoxProfile;
  }
  </code></pre>
  <p>With this source code I can not find the div element with the information I want to extract.</p>
  <p>It would be really great, if somebody could help me with this.</p>