What I wish someone would have told me about using RabbitMQ (2020)
I&#x27;d say this is a good introduction into reasons why self hosting clustered software is not something to do lightly. He&#x27;s basically running rabbit mq on Windows. Yikes. And also, why? I&#x27;m sure it can be done responsibly. But allowing all nodes to self update and reboot randomly sounds like amateur hour to me (that actually happened apparently).<p>So, don&#x27;t do that. I don&#x27;t use rabbit mq currently but we do use Elasticsearch, which has similar clustering capability and used to be more susceptible to split brain situations (been there, dealt with that)<p>These days, I recommend using Elastic Cloud and avoid self hosting it. It&#x27;s only cheaper until the first time you have to deal with a split brain cluster because you botched an update, mis-configured it, etc. One of the nice features in Elastic cloud is that you can click an update button and it will orchestrate a rolling restart. If you don&#x27;t know what that is, you should not be operating a cluster of any kind.<p>I&#x27;m sure there are similar cloud based services for rabbitmq. Probably well worth the money unless your in house ops team is super experienced with operating it (which clearly wasn&#x27;t the case here). Such a team would cost you many hundreds of thousands of dollars per year. One person does not cut it. You need at least a 3 or 4 so you can afford them taking vacations, sick leave, leaving, or dying in some tragic accident. Half a million pays for some pretty nice cloud based clustering capacity. A good team will cost you more.<p>For reference, we pay about 70&#x2F;month for a tiny Elastic cloud search cluster that is actually good enough. I can double the price and capacity with a simple slider and it would still be cheap. One hour of my time is more than that with my normal freelance rate. My monthly rate would pay for an enormous cluster that far exceeds anything we need and it would still be cheaper than making sure we have four people with my skill set in the team (we don&#x27;t) able and willing to look after it at all hours. The largest cluster I&#x27;ve ever dealt with was a self hosted monster that could index billions of documents per hour (millions per second). You so much as looked wrong at it, all hell would break loose. That cluster was one of several managed by a very experienced ops team that probably cost millions per years. That&#x27;s the price of doing business at scale when self hosting.<p>Most companies running into trouble with ES cut corners on doing it right and then pay the price with preventable outages, scaling issues, technical debt, etc. Cloud based services don&#x27;t completely prevent this but if you know what you are doing, they provide a nice level of safety and risk mitigation.