I noticed that I sometimes get blocked while scraping because of a session cookie being used on too many pages.
Is there a way to simply clear all cookies completely during crawling to get back to the initial state of the crawler?
Facing the similar situation myself. I can get away easily here, but one idea that I have is to subclass CookieMiddleware and then write a method to tweak jar variable directly. It's dirty, but maybe is worth consideration.
Another option would be to write a feature request to at least have a function to clear the cookies. Can easily take another year to implement, if deemed needed at all, I don't particularly trust scrapy devs here.
Just occured to me, that you can use your own cookiejar meta, and if you want to return to the clean state, you simply use different value (something like incrementing an integer would do).
Clear cookies on scrapy completely instead of changing them