Scrapy ROBOTSTXT_OBEY not working in all cases
Scrapy uses Python standard robots.txt parser which doesn't support wildcards.
So I have a Scrapy project set up and I have enabled the ROBOTSTXT_OBEY middleware which is working fine on robot.txt files that are in the following format:
User-agent: *
Disallow: /tools/
But when the same spider on a site with a robots.txt file in the following format it doesn't work:
User-agent: *
Disallow: *?next
This results in pages still being crawled that should be blocked by robots.txt, the by the way in completely valid markup for a robots.txt file.
Just wondered if anyone could shed any light on why this might be?