How we can get List of urls after crawling website from scrapy in costom python script?
You can use a file to pass the urls from scrapy to your python script.
Or you can print the urls with a mark in your scrapy, and use your python script to catch the stdout of your scrapy.Then parse it to list.
I am working with a script where i need to crawl websites, need to crawl only base_url site. Anyone who has pretty good idea how i can launch scarpy in custom python scripts and get urls link in list?