How does xpathselector affect the speed of scrapy crawl running?
This has nothing to do with download speed.
XPath //* selects the entire page. XPath //script/text() selects only text inside script elements. So of course the second one is faster, because there is less text to search with the re() call!
I am using Scrapy crawler to crawl over 100k pages website.  The speed is the big concern in this case.  Today I noticed that hxs.select('//*').re('something') is way slower than hxs.select('//script/text()').re('something').  Can any expert explain to me why? 
As I understand, the crawler should download the entire page no matter what xpath selector I use.  So the xpath should not affect the speed much at all.
Thanks a lot for any tips.