title: crawl site that has infinite scrolling using python
body: <p>I have been doing research and so far I found out the python package that I will plan on using its <a href="http://scrapy.org/" rel="noreferrer">scrapy</a>, now I am trying to find out what is a good way to build a scraper using scrapy to crawl site with infinite scrolling. After digging around I found out that there is a package call selenium and it has python module. I have a feeling someone has already done that using Scrapy and <a href="https://pypi.python.org/pypi/selenium" rel="noreferrer">Selenium</a> to scrape site with infinite scrolling. It would be great if someone can point towards to an example. </p>