I am using Scrapy crawler to crawl over 100k pages website.  The speed is the big concern in this case.  Today I noticed that hxs.select('//*').re('something') is way slower than hxs.select('//script/text()').re('something').  Can any expert explain to me why? 
As I understand, the crawler should download the entire page no matter what xpath selector I use.  So the xpath should not affect the speed much at all.
Thanks a lot for any tips.
I am afraid that you might look for 'something' in the entire document, so you probably should still use hxs.select('//*').re('something').
And about the speed question: the answer is that if you look for the word 'something' in a document which is 4k large, of course it will take longer then filtering the document for text() and after looking for that word within that text.
How does xpathselector affect the speed of scrapy crawl running?