title: Web scraping: Automating button click
body: <p>I am trying to scrape data off a website using <a href="http://scrapy.org/" rel="nofollow noreferrer">Scrapy</a>, a python framework. I can get the data from the website using the spiders but the problem occurs when I try to navigate through the website.</p>
  
  <p>According to <a href="https://stackoverflow.com/questions/6682503/click-a-button-in-scrapy">this</a> post Scrapy does not handle Javascript well.</p>
  
  <p>Also, as stated in the accepted answer, I cannot use mechanize or lxml. It suggests using a combination of <code>Selenium and Scrapy</code>.</p>
  
  <p>Function of the button:</p>
  
  <blockquote>
    <p>I am browsing through offers on  a website. The function of the button
    is to show more offers. SO on clicking it, it calls a javascript
    function which loads the results.</p>
  </blockquote>
  
  <p>I also was looking at <code>CasperJS and PhantomJS</code>. Will they work?</p>
  
  <p>I just need to automate the clicking of a button. How do I go about this?</p>