I am using scrapy to crawl a site which seems to be appending random values to the query string at the end of each URL. This is turning the crawl into a sort of an infinite loop.
How do i make scrapy to neglect the query string part of the URL's?
There is a function url_query_cleaner in w3lib.url module (used by scrapy itself) to clean urls keeping only a list of allowed arguments.
How do I remove a query from a url?