how to use scrapy-redis pipeline?
The pipeline is a different script, yes. In the settings file you can enable the pipeline. A pipeline can be used to store the crawled results in any database you want.
I am using scrapy-redis now, and I am ok with it, and I am success to crawl in different computer by using the same redis server.
But I don't understand how to use the scrapy-redis pipeline properly.
In my understanding, I think I need another script than the spiders to deal with the item in the redis pipeline list, then I can do stuffs like store them into the database.
Do I understand right, do I have to write another script, which is somehow dependent from the spider?