I am working with a script where i need to crawl websites, need to crawl only base_url site. Anyone who has pretty good idea how i can launch scarpy in custom python scripts and get urls link in list?
You can use a file to pass the urls from scrapy to your python script.
Or you can print the urls with a mark in your scrapy, and use your python script to catch the stdout of your scrapy.Then parse it to list.
How we can get List of urls after crawling website from scrapy in costom python script?