Answers: <p>I would set the <code>leave_on_terminate</code> option to true. As per the <a href="https://www.consul.io/docs/agent/options.html#leave_on_terminate" rel="noreferrer">documentation</a></p>
  
  <blockquote>
    <p><code>leave_on_terminate</code> If enabled, when the agent receives a TERM signal,
    it will send a Leave message to the rest of the cluster and gracefully
    leave. The default behavior for this feature varies based on whether
    or not the agent is running as a client or a server (prior to Consul
    0.7 the default value was unconditionally set to false). On agents in client-mode, this defaults to true and for agents in server-mode, this
    defaults to false.</p>
  </blockquote>
  
  <p>What happens when a node is shutdown gracefully is to send SIGTERM to all processes before shutdown, with this setting on the consul agent will leave the cluster so it won't be considered as a node which can restart and be back in the cluster in a few hours (which is what your quote says it does by default).</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-11-20T01:05:27.247Z
Answers: With this turned on, does the dead client consul get reaped right away or is there still a delay? I've tested this option on client-mode Consul and after running `shutdown -h now`, dead node still shows up...
Answers: Casper
Answers: CC BY-SA 4.0
Answers: 2019-11-20T06:16:50.097Z
Answers: @Casper there's various way this may   not work as expected, I guess it will depend if your daemon system let enough time to the consul-client to stop gracefully (assuming you didn't launch the client with a daemon manager and not as a command)
Answers: Tensibai
Answers: CC BY-SA 4.0
Answers: 2019-11-20T18:24:26.920Z
Answers: Thanks for replying, it's been a while :) So my goal is to have a low reap time for dead nodes, the default 72 hours is too long and it seems like there is no way to customize reap time on client nodes
Answers: Casper
Answers: CC BY-SA 4.0
Answers: 2019-11-20T22:18:36.320Z
Answers: @Casper as I said above, we'd need more details to give any advice, in a systemd setup the stop part could be tweaked to let consul client stop properly before continuing the shutdown process so it could remove itself, or there's a bug somewhere, but with the current information we can only guess and SE sites are not well adapted for this kind of debug
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2017-03-01T16:09:50.427Z
Answers: 2017-03-01T16:09:50.427Z
Answers: Tensibai
Body: <p>We have auto-scaling Docker environments in which we use Consul for service discovery. These environments can add or remove one instance every few minutes.</p>
  
  <p>Our early Consul testing showed that it was very easy for Consul to loose its quorum. Perhaps naively, our very first experimentation was a setup in which we would start a Consul server on all instances and have that Consul server join the cluster. That part was working fine.</p>
  
  <p>However, Consul does not reap unreachable nodes quickly (it take about 72 hours?) in a very scalable environments that means that the list of Consul servers keeps growing and over time, most of them are "unreachable" and at that point, the cluster loses its quorum.</p>
  
  <p>We've seen armon's response from almost two years ago on this issue on GitHub: <a href="https://github.com/hashicorp/consul/issues/454#issuecomment-125767550" rel="noreferrer">https://github.com/hashicorp/consul/issues/454#issuecomment-125767550</a></p>
  
  <blockquote>
    <p>Most of these problems are caused by our default behavior of
    attempting a graceful leave. Our mental model is that servers are long
    lived and don't shutdown for any reason other than unexpected power
    loss, or a graceful maintenance in which case you need to leave the
    cluster. In retrospect that was a bad default. Almost all of this can
    be avoided by just kill -9 the Consul server, in affect simulating
    power loss.</p>
  </blockquote>
  
  <p>We were trying to avoid running dedicated, long-lived nodes. Keep in mind that at no point, we remove N/2+1 instances from an auto-scaling group. The EC2 cluster is able at any point in time to reach most of the nodes and should be able to vote whether a node should be removed from the Consul (or other tool) cluster.</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-03-01T16:20:39.650Z
Comments: I would imagine that a meaningful answer is not possible without more data, such as: 
  
  How big (in numbers) is your instance population? 
  
  Did you attempt to debug the underlying quorum/consensus mechanism to see what causes the delays in reaping inactive members? 
  
  What is the timing of instance removal, did you track if the consul instance actually has time to send word of its demise (graceful or not) to the rest of the ASG?
Comments: Michael Bravo
Comments: CC BY-SA 3.0
Comments: 2017-03-01T17:15:25.193Z
Comments: Are you starting them as a "server-mode" or as an "client-mode"?  The leave_on_terminate documentation says that "client-mode" will default to true.  To me it seems like consul agents started as "server-mode" should be longer lived than you describe
Comments: Thymine
Comments: CC BY-SA 3.0
Comments: 2017-03-01T19:29:56.253Z
Comments: Thank you all. Tensibai's answer is what we were looking for.
Comments: Alexandre
Tags: <docker><autoscaling><consul>
Title: How to manage Consul and its quorum in auto-scaling environments?
UserDisplayName: Alexandre
ThreadText: # How to manage Consul and its quorum in auto-scaling environments?
  
  (Asked by: username_0 on 2017-03-01)
  
  We have auto-scaling Docker environments in which we use Consul for service discovery. These environments can add or remove one instance every few minutes.
  
  Our early Consul testing showed that it was very easy for Consul to loose its quorum. Perhaps naively, our very first experimentation was a setup in which we would start a Consul server on all instances and have that Consul server join the cluster. That part was working fine.
  
  However, Consul does not reap unreachable nodes quickly (it take about 72 hours?) in a very scalable environments that means that the list of Consul servers keeps growing and over time, most of them are "unreachable" and at that point, the cluster loses its quorum.
  
  We've seen armon's response from almost two years ago on this issue on GitHub: <https://github.com/hashicorp/consul/issues/454#issuecomment-125767550>
  
  > Most of these problems are caused by our default behavior of
  > attempting a graceful leave. Our mental model is that servers are long
  > lived and don't shutdown for any reason other than unexpected power
  > loss, or a graceful maintenance in which case you need to leave the
  > cluster. In retrospect that was a bad default. Almost all of this can
  > be avoided by just kill -9 the Consul server, in affect simulating
  > power loss.
  
  We were trying to avoid running dedicated, long-lived nodes. Keep in mind that at no point, we remove N/2+1 instances from an auto-scaling group. The EC2 cluster is able at any point in time to reach most of the nodes and should be able to vote whether a node should be removed from the Consul (or other tool) cluster.
  
  **Tags:** docker, autoscaling, consul
  
  > **Comments:**
  > **username_1**: I would imagine that a meaningful answer is not possible without more data, such as:   How big (in numbers) is your instance population?   Did you attempt to debug the underlying quorum/consensus mechanism to see what causes the delays in reaping inactive members?   What is the timing of instance removal, did you track if the consul instance actually has time to send word of its demise (graceful or not) to the rest of the ASG?  
  > **username_2**: Are you starting them as a "server-mode" or as an "client-mode"?  The leave_on_terminate documentation says that "client-mode" will default to true.  To me it seems like consul agents started as "server-mode" should be longer lived than you describe  
  > **username_0**: Thank you all. username_3's answer is what we were looking for.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2017-03-01. Score: 6 | Accepted answer
  
  I would set the `leave_on_terminate` option to true. As per the [documentation](https://www.consul.io/docs/agent/options.html#leave_on_terminate)
  
  > `leave_on_terminate` If enabled, when the agent receives a TERM signal,
  > it will send a Leave message to the rest of the cluster and gracefully
  > leave. The default behavior for this feature varies based on whether
  > or not the agent is running as a client or a server (prior to Consul
  > 0.7 the default value was unconditionally set to false). On agents in client-mode, this defaults to true and for agents in server-mode, this
  > defaults to false.
  
  What happens when a node is shutdown gracefully is to send SIGTERM to all processes before shutdown, with this setting on the consul agent will leave the cluster so it won't be considered as a node which can restart and be back in the cluster in a few hours (which is what your quote says it does by default).
  
  > **Comments:**
  > **username_4**: With this turned on, does the dead client consul get reaped right away or is there still a delay? I've tested this option on client-mode Consul and after running `shutdown -h now`, dead node still shows up...  
  > **username_3**: @username_4 there's various way this may   not work as expected, I guess it will depend if your daemon system let enough time to the consul-client to stop gracefully (assuming you didn't launch the client with a daemon manager and not as a command)  
  > **username_4**: Thanks for replying, it's been a while :) So my goal is to have a low reap time for dead nodes, the default 72 hours is too long and it seems like there is no way to customize reap time on client nodes  
  > **username_3**: @username_4 as I said above, we'd need more details to give any advice, in a systemd setup the stop part could be tweaked to let consul client stop properly before continuing the shutdown process so it could remove itself, or there's a bug somewhere, but with the current information we can only guess and SE sites are not well adapted for this kind of debug  
  
  ---
  
  