I am using Scrapy crawler to crawl over 100k pages website.  The speed is the big concern in this case.  Today I noticed that hxs.select('//*').re('something') is way slower than hxs.select('//script/text()').re('something').  Can any expert explain to me why? 
As I understand, the crawler should download the entire page no matter what xpath selector I use.  So the xpath should not affect the speed much at all.
Thanks a lot for any tips.
This has nothing to do with download speed.
XPath //* selects the entire page. XPath //script/text() selects only text inside script elements. So of course the second one is faster, because there is less text to search with the re() call!
How does xpathselector affect the speed of scrapy crawl running?