Sounds like I have some more research to do - thanks for the detailed response.<p>Btw, you were right, I meant scrapy-cluster and not scrapyd-cluster: <a href="https:&#x2F;&#x2F;github.com&#x2F;istresearch&#x2F;scrapy-cluster" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;istresearch&#x2F;scrapy-cluster</a>. There is a requirement on Zookeeper&#x2F;Kafka, which is the main showstopper for me.