Usa parameter as collection name in a scrapy project
I've got an answer.
for example:
using -s collection_name=abc for scrapy crawl command, then get the parameter in pipelines.py using param = settings.get('collection_name').
This is also found in stackoverflow, but can't remember which ticket.
Hope this would help some one facing same problem.
The spider is to crawl info on a certain B2B website, and I want it to be a webserver, where user sumbit a url then the spider starts crawl.
The url seems like: apple.b2bxxx.com, which is a minisite on a B2B website, where all the products are listed. The "apple" might be different because different companies use different names for there minisite, and duplication is not allowed.
On the backend, it's MongoDB to store the data scraped.
What I have done, is that, I can collect info on the given url, but all data are stored in the same db.collection.
I know I can get parameters using "-a" for running scrapy, but how should I use it?
Should I change the pipelines.py or the spider python file?
Any suggestions?