How does xpathselector affect the speed of scrapy crawl running?
XPath definitely has a role in crawler speed, crawler download the page, but Xpath process the Html that crawler downloaded. so if the page is big then xpath will take time to process whole Html.
I am using Scrapy crawler to crawl over 100k pages website.  The speed is the big concern in this case.  Today I noticed that hxs.select('//*').re('something') is way slower than hxs.select('//script/text()').re('something').  Can any expert explain to me why? 
As I understand, the crawler should download the entire page no matter what xpath selector I use.  So the xpath should not affect the speed much at all.
Thanks a lot for any tips.