Does Scrapy ping the website every time I use response.xpath? Or is there one response value stored in memory per request and all subsequent xpath queries are run locally?
Response is stored in memory. That said, every time you call response.xpath it doesn't hit the website on the matter but memory.
Optimizing Scrapy xpath queries