I'm currently trying to create a spider which crawls each result and takes some info from each of them. The only problem is that I don't know how to find the URL that I'm currently on (I need to retrieve that too).
Is there any way to do that?
I know how to do that using Selenium and Scrapy-Selenium, but I'm only using a simple CrawlSpider for this project.
You can use:
current_url = response.request.url
Is there any way to find the URL that you are currently scraping?