title: Use phantomJS in R to scrape page with dynamically loaded content
body: <p><strong>Background</strong>
  I'm currently scraping product information from some websites in R using rvest. This works on all but one website where the content seems to be loaded dynamically via angularJS (?), so cannot be loaded iteratively e.g. via URL parameters (as I did for other websites). The specific url is as follows:</p>
  
  <p><a href="http://www.hornbach.de/shop/Badarmaturen/Waschtischarmaturen/S3584/artikelliste.html" rel="noreferrer">http://www.hornbach.de/shop/Badarmaturen/Waschtischarmaturen/S3584/artikelliste.html</a></p>
  
  <p>Please keep in mind I don't have admin rights on my machine and can only implement solutions that require either no or only single-time granting of admin rights</p>
  
  <p><strong>Desired Output</strong>
  In the end a table in R with product information (e.g. label, price, rating) => In this question, though, I purely need help to dynamically load and store the website; I can handle the postprocessing in R on my own.
  It'd be absolutely great if you could push me in the right direction; maybe one of my approaches listed below are on the right track, but I just seem unable to transfer those to the stated website.</p>
  
  <p><strong>Current approach</strong>
  I found phantomJS as a headless browser that afaik should be able to handle this issue. I have close to none knowledge of Java Script at all and syntax differs (at least for me) heavily from languages I'm more used to (R, Matlab, SQL) that I really struggle to implement approaches suggested somewhere else that might work in my code.
  Based on <a href="https://gist.github.com/flovv/91453712e8a6ba957e63" rel="noreferrer">this example</a> (thanks a lot) I managed to retrieve at least information from the first shown page with this code:</p>
  
  <p>R:</p>
  
  <pre><code>require(rvest)
  
  ## change Phantom.js scrape file
  url &lt;- 'http://www.hornbach.de/shop/Badarmaturen/Waschtischarmaturen/S3584/artikelliste.html'
  
  lines &lt;- lines &lt;- readLines("scrape_final.js")
  lines[1] &lt;- paste0("var url ='", url ,"';")
  writeLines(lines, "scrape_final.js")
  
  ## Download website
  system("phantomjs scrape_final.js")
  
  ### use Rvest to scrape the downloaded website.
  web &lt;- read_html("1.html")
  content &lt;- html_nodes(web, 'div.paging-indicator')# %&gt;% html_attr('href')
  content &lt;- html_text(content) %&gt;% as.data.frame()
  </code></pre>
  
  <p>and the corresponding PhantomJS script:</p>
  
  <pre><code>var url ='http://www.hornbach.de/shop/Badarmaturen/Waschtischarmaturen/S3584/artikelliste.html';
  var page = new WebPage() 
  var fs = require('fs'); 
  
  page.open(url, function (status) { 
           just_wait(); 
  }); 
  
  
  function just_wait() { 
      setTimeout(function() { 
                fs.write('1.html', page.content, 'w'); 
             phantom.exit(); 
      }, 2500); 
  }  
  </code></pre>
  
  <p><strong>what does not work // research</strong>
  While this code retrieves information from the first page, I do need to iterate through all x product pages. I tried to extend the code above with the following examples:</p>
  
  <p><a href="https://stackoverflow.com/questions/33146879/scrape-dynamic-loading-">Unable to scrape multiple pages using phantomjs in r</a></p>
  
  <p>[Scrape dynamic loading pages with phantomjs][3]</p>
  
  <p>[Web scraping dynamically loading data in R][4]</p>
  
  <p>The examples led me to the idea </p>
  
  <ul>
  <li>either clicking the "next page" button</li>
  <li><p>or somehow inject the correct pagination value</p>
  
  <ol>
  <li><strong>Either click on the "next page" button</strong></li>
  </ol></li>
  </ul>
  
  <p>This looks like the following</p>
  
  <pre><code>    var url ='http://www.hornbach.de/shop/Badarmaturen/Waschtischarmaturen/S3584/artikelliste.html';
      var page = require('webpage').create(); 
      var fs = require('fs'); 
  
      page.open(url, function (status) { 
      age.open(url, function() {
        page.includeJs("http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js", function() {
          page.evaluate(function() {
            $("paging-btn right").click();
              just_wait(); 
          });
          phantom.exit()
        });
  
      }); 
  
  
      function just_wait() { 
          setTimeout(function() { 
                    fs.write('1.html', page.content, 'w'); 
                 phantom.exit(); 
          }, 2500); 
      } 
  </code></pre>
  
  <p>But that doesn't get me anywhere due to poor syntax and maybe other things. 
  Calling this script from R doesn't produce an error unfortunately, it just runs for ages so I have to quit it (while the working script only takes few secs).</p>
  
  <p>I used the gadget inspector from firefox to retrieve the button name, but also that might be wrong:</p>
  
  <pre><code>&lt;a class="paging-btn right rel="next" ng-click="goToNextPage()" 
  ng-hide="articleData.pageNumber == articleData.pageCount" 
  href="javascript:void(0);"&gt;right&lt;/a&gt; 
  </code></pre>
  
  <ol start="2">
  <li><strong>change the pagination parameter on load</strong></li>
  </ol>
  
  <p>I tried to workon the given example here
  <a href="https://stackoverflow.com/questions/15356219/passing-variable-into-page-evaluate-phantomjs">Passing variable into page.evaluate - PhantomJS</a></p>
  
  <p>but also just got a script that never ended in R</p>
  
  <p><strong>Additional notes</strong>
  It looks like I'm only allowed to post two links, so unfortunately I couldn't link all sources I've researched and tested.</p>
  
  <p>I'm well aware this is huge and messy info at once and if you can help me to improve/better structure my question, please let me know. I'll do my best to be responsive and get you anything you need to assist.</p>