Answers: <p>Consider using the <a href="https://github.com/rabbitmq/rabbitmq-autocluster" rel="nofollow noreferrer">rabbitmq/rabbitmq-autocluster</a> plugin:</p>
  
  <blockquote>
    <p>A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:</p>
    
    <ul>
    <li><a href="https://www.consul.io/" rel="nofollow noreferrer">Consul</a>,</li>
    <li><a href="https://github.com/coreos/etcd" rel="nofollow noreferrer">etcd2</a></li>
    <li>DNS A records</li>
    <li>AWS EC2 tags</li>
    <li>AWS Autoscaling Groups</li>
    </ul>
  </blockquote>
  
  <p>There is a <a href="https://github.com/aweber/rabbitmq-autocluster/wiki/AWS%20Configuration" rel="nofollow noreferrer">fair bit of configuration</a> to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.</p>
  
  <p>If you were to use AWS Autoscaling Groups then you would add the following to your <code>rabbitmq.config</code>:</p>
  
  <pre><code>[
    {rabbit, [ ... ]},
    {autocluster, [
      {backend, aws},
      {aws_autoscaling, true},
      {aws_ec2_region, "us-west-2"}
    ]}
  ].
  </code></pre>
  
  <p>If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:</p>
  
  <pre><code>[
    {rabbit, [ ... ]},
    {autocluster, [
      {backend, aws},
      {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
      {aws_ec2_region, "us-east-1"},
      {aws_access_key, "..."},
      {aws_secret_key, "..."}
    ]}
  ].
  </code></pre>
  
  <p>With all of that said I strongly recommend using <a href="https://www.consul.io/" rel="nofollow noreferrer">Consul by HashiCorp</a> as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-03-25T22:06:11.887Z
Answers: thank you for the detailed explanation and the diagram, so AFAIK `rabbitmq/rabbitmq-autocluster plugin` will know also to remove node from the replication list once the node is down, one more thing if may I ask, I thought to start with `2-node` cluster, are you suggest to start with `3-node` cluster as you describe in your diagram with policy  ` rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}' ` ? or should I post it in a another question ?
Answers: Berlin
Answers: CC BY-SA 3.0
Answers: 2017-03-26T08:47:05.147Z
Answers: Possibly another question as the formatting in comments isn't all that easy to use.  I have always advised clients when adopting cloud to use larger number of small instances rather than the inverse.  This is because failure is expected in the cloud, if you have a 2-node cluster and 1 fails you have lost 50% of your capacity, whereas if you have a 3-node cluster and 1 fails you only lose 33%.
Answers: Richard Slater
Answers: CC BY-SA 3.0
Answers: 2017-04-01T15:13:36.257Z
Answers: I have configured my cluster using `rabbitmq/rabbitmq-autocluster plugin` and it is work pretty well however when node goes down RabbitMQ does not remove it from the replication list, any idea why ?
Answers: Berlin
Answers: CC BY-SA 3.0
Answers: 2017-04-01T16:18:01.497Z
Answers: Have you enabled the Cluster Cleanup configuration option?
Answers: Richard Slater
Answers: CC BY-SA 3.0
Answers: 2017-04-01T22:50:28.037Z
Answers: thanks, found it `https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings`, I will try that.
Answers: Berlin
Answers: CC BY-SA 4.0
Answers: 2019-10-11T12:26:25.887Z
Answers: Links are great https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings
Answers: Oliver Salzburg
Answers: CC BY-SA 3.0
Answers: 2017-03-25T18:47:05.343Z
Answers: 2017-03-25T18:47:05.343Z
Answers: Richard Slater
Body: <p>I am planning to create RabbitMQ cluster using Ansible on AWS VPC with Amazon internal load balancer as the frontend to point connections to it.</p>
  
  <p>Any suggestion how to remove a dead node from RabbitMQ cluster based on autoscaling rule where nodes can go up and down, or if you are using spot instances?</p>
  
  <p>When a node goes down, RabbitMQ does not remove it from the replication list automatically, I can see <code>Node not running</code> in the management UI.</p>
  
  <p>I managed to join to the cluster a scaled instance automatically via Ansible and userdata.</p>
  
  <p><a href="https://i.sstatic.net/2O2Tx.png" rel="noreferrer"><img src="https://i.sstatic.net/2O2Tx.png" alt="Diagram of Infrastructure"></a></p>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-25T15:33:23.517Z
Comments: @Pierre.Vriens, I changed to only 1 question, thanks.
Comments: Berlin
Comments: CC BY-SA 3.0
Comments: 2017-03-25T15:35:09.747Z
Comments: Merci! You didn't destroy the other 4 (or so) questions I hope. Maybe keep them as folowup question, to the extend they are still relevant?
Comments: Pierre.Vriens
Comments: CC BY-SA 3.0
Comments: 2017-03-25T15:39:20.067Z
Comments: Yep, the questions are still relevant, but this one is the most important. I will post another question later :) thanks!
Comments: Berlin
Comments: CC BY-SA 3.0
Comments: 2017-03-25T18:59:45.530Z
Comments: @Berlin I drew a diagram that represents what I think you are describing, if you meant something else then please do let me know and I will adapt.
Comments: Richard Slater
Comments: CC BY-SA 3.0
Comments: 2017-03-25T19:45:42.730Z
Comments: Hey @Pierre.Vriens - I don't mind I had a few minutes and wanted to clarify my assumption, theoretically, I could add it to my answer, and I may well do that.
Comments: Richard Slater
Tags: <amazon-web-services><ansible><aws-vpc><rabbitmq>
Title: How to automatically remove a dead node from RabbitMQ cluster
UserDisplayName: Berlin
ThreadText: # How to automatically remove a dead node from RabbitMQ cluster
  
  (Asked by: username_0 on 2017-03-25)
  
  I am planning to create RabbitMQ cluster using Ansible on AWS VPC with Amazon internal load balancer as the frontend to point connections to it.
  
  Any suggestion how to remove a dead node from RabbitMQ cluster based on autoscaling rule where nodes can go up and down, or if you are using spot instances?
  
  When a node goes down, RabbitMQ does not remove it from the replication list automatically, I can see `Node not running` in the management UI.
  
  I managed to join to the cluster a scaled instance automatically via Ansible and userdata.
  
  [![Diagram of Infrastructure](https://i.sstatic.net/2O2Tx.png)](https://i.sstatic.net/2O2Tx.png)
  
  **Tags:** amazon-web-services, ansible, aws-vpc, rabbitmq
  
  > **Comments:**
  > **username_0**: @username_1, I changed to only 1 question, thanks.  
  > **username_1**: Merci! You didn't destroy the other 4 (or so) questions I hope. Maybe keep them as folowup question, to the extend they are still relevant?  
  > **username_0**: Yep, the questions are still relevant, but this one is the most important. I will post another question later :) thanks!  
  > **username_2**: @username_0 I drew a diagram that represents what I think you are describing, if you meant something else then please do let me know and I will adapt.  
  > **username_2**: Hey @username_1 - I don't mind I had a few minutes and wanted to clarify my assumption, theoretically, I could add it to my answer, and I may well do that.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-03-25. Score: 4 | Accepted answer
  
  Consider using the [rabbitmq/rabbitmq-autocluster](https://github.com/rabbitmq/rabbitmq-autocluster) plugin:
  
  > A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:
  >
  > * [Consul](https://www.consul.io/),
  > * [etcd2](https://github.com/coreos/etcd)
  > * DNS A records
  > * AWS EC2 tags
  > * AWS Autoscaling Groups
  
  There is a [fair bit of configuration](https://github.com/aweber/rabbitmq-autocluster/wiki/AWS%20Configuration) to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.
  
  If you were to use AWS Autoscaling Groups then you would add the following to your `rabbitmq.config`:
  
  ```
  [
    {rabbit, [ ... ]},
    {autocluster, [
      {backend, aws},
      {aws_autoscaling, true},
      {aws_ec2_region, "us-west-2"}
    ]}
  ].
  
  ```
  
  If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:
  
  ```
  [
    {rabbit, [ ... ]},
    {autocluster, [
      {backend, aws},
      {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
      {aws_ec2_region, "us-east-1"},
      {aws_access_key, "..."},
      {aws_secret_key, "..."}
    ]}
  ].
  
  ```
  
  With all of that said I strongly recommend using [Consul by HashiCorp](https://www.consul.io/) as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.
  
  > **Comments:**
  > **username_0**: thank you for the detailed explanation and the diagram, so AFAIK `rabbitmq/rabbitmq-autocluster plugin` will know also to remove node from the replication list once the node is down, one more thing if may I ask, I thought to start with `2-node` cluster, are you suggest to start with `3-node` cluster as you describe in your diagram with policy  ` rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}' ` ? or should I post it in a another question ?  
  > **username_2**: Possibly another question as the formatting in comments isn't all that easy to use.  I have always advised clients when adopting cloud to use larger number of small instances rather than the inverse.  This is because failure is expected in the cloud, if you have a 2-node cluster and 1 fails you have lost 50% of your capacity, whereas if you have a 3-node cluster and 1 fails you only lose 33%.  
  > **username_0**: I have configured my cluster using `rabbitmq/rabbitmq-autocluster plugin` and it is work pretty well however when node goes down RabbitMQ does not remove it from the replication list, any idea why ?  
  > **username_2**: Have you enabled the Cluster Cleanup configuration option?  
  > **username_0**: thanks, found it `https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings`, I will try that.  
  > **username_3**: Links are great https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings  
  
  ---
  
  