Python 3 comes to Scrapy
I mentioned some of the issues I had with scrapy ages ago on reddit: <a href="http:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;Python&#x2F;comments&#x2F;g112q&#x2F;installing_and_using_scrapy_web_crawler_to_search&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;Python&#x2F;comments&#x2F;g112q&#x2F;installing_and...</a><p>Never got a reply, but I&#x27;ll reproduce here; I wonder how much of this is still true?<p>&quot;
The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started.<p>When I used it, I wanted to scrape a site until certain conditions were met; when the last page scraped returned no objects. I wanted all results initially returned from a page to be dropped if they were older than a certain date; Thus I wanted Scrapy to keep scraping until no new items were found. Also, I wanted the latest date of the items returned so I could use this the next time I scrape.<p>I created the &#x27;DropElderMiddleware&#x27; middleware to do this. I couldn&#x27;t see any other way of making calculations based on items returned from a particular page.<p>I could never figure out what the difference between input and output processors where, or when I should use one or the other.<p>The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain.<p>The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can&#x27;t remember the details of why I found this hard, but I can try to come up with an example if you like?<p>In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy&#x27;s features I didn&#x27;t need e.g. I didn&#x27;t want command-line control (I would actually prefer not to use the interface, though didn&#x27;t discover how I could write a python script to apply the spider directly).<p>Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy&#x27;s object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what&#x27;s going on in my scripts.
&quot;