Answers: <p>As a best practice, you should gracefully deregister the node gracefully. In this case, consul will know the the node has left and will be removed from the cluster. Otherwise, the consul cannot distinguish between a temporary failure, agent crash, network partition, etc.</p>
  
  <p>There is a github <a href="https://github.com/hashicorp/consul/issues/544" rel="nofollow noreferrer">issue</a> as well related to this. </p>
  
  <p>Quoting few important points from the issue.</p>
  
  <blockquote>
    <p>The nodes should automatically reap out after 72hours (not yet configurable, but soon). Otherwise, the best route is to issue a graceful leave before destroying the nodes (consul leave), so that they can be reaped immediately. They are kept around for that long since without a graceful leave, Consul cannot distinguish between a temporary failure, agent crash, network partition, etc.</p>
  </blockquote>
  
  <p>.</p>
  
  <blockquote>
    <p>force-leave should push them into the "left" state. Nodes are not reaped until they are in failed for 72h or in the left state. "force-leave" just moves a node from the "failed" -> "left" state. They are not removed from the members list for 24 or 72h.</p>
  </blockquote>
  
Answers: CC BY-SA 4.0
Answers: 2019-11-20T21:57:09.807Z
Answers: What I'm trying to figure out is that is there a configurable setting for reap time on the clients (for now only Consul servers have these settings). Gracefully leave is no longer sustainable for cloud environment anymore since VMs are very redundant and nobody cares whether a node is shutdown.
Answers: Casper
Answers: CC BY-SA 4.0
Answers: 2019-11-21T14:28:20.180Z
Answers: As of now, this property is not available for agents. There is a github issue for enabling this. Please refer https://github.com/hashicorp/consul/issues/2982
Answers: Samit
Answers: CC BY-SA 4.0
Answers: 2021-09-01T13:10:49.840Z
Answers: had to do the ugly force-leave command also to remove federation between 2 consul (VM - K8s) | 
  also doing the -prune will give me "the flag provided but not defined: -prune"
  | consul version 1.1.0
Answers: Madox
Answers: CC BY-SA 4.0
Answers: 2023-01-15T16:39:30.310Z
Answers: @Casper Would you be able to accept this answer if this had fixed the issue? This will help other people.
Answers: Samit
Answers: CC BY-SA 4.0
Answers: 2019-11-20T19:07:42.563Z
Answers: 2019-11-20T19:07:42.563Z
Answers: Samit
Body: <p>Background: When running Consul client-mode in AWS, EC2s are constantly getting replaced, which results in a lot of dead client nodes when looking at the Consul UI or via <code>consul members</code>.</p>
  
  <p>Already tried to enabled <code>reconnect_timeout</code>, <code>leave_on_terminate</code> (default to true on Client but tried to turn on anyway), and Autopilot <code>cleanup_dead_servers</code> to no avail.</p>
  
  <p>The delay seems to be inconsistent as well, some got reaped after 4~7 days, some are never reaped, <code>consul force-leave</code> had to be used.</p>
  
Comments: CC BY-SA 4.0
Comments: 2019-11-20T19:00:09.260Z
Comments: which consul version you are using?
Comments: Samit
Comments: CC BY-SA 4.0
Comments: 2019-11-20T21:51:36.753Z
Comments: Both servers and clients are on 1.0.0
Comments: Casper
Tags: <linux><consul>
Title: Consul: how to remove dead client nodes quickly
UserDisplayName: Casper
ThreadText: # Consul: how to remove dead client nodes quickly
  
  (Asked by: username_0 on 2019-11-20)
  
  Background: When running Consul client-mode in AWS, EC2s are constantly getting replaced, which results in a lot of dead client nodes when looking at the Consul UI or via `consul members`.
  
  Already tried to enabled `reconnect_timeout`, `leave_on_terminate` (default to true on Client but tried to turn on anyway), and Autopilot `cleanup_dead_servers` to no avail.
  
  The delay seems to be inconsistent as well, some got reaped after 4~7 days, some are never reaped, `consul force-leave` had to be used.
  
  **Tags:** linux, consul
  
  > **Comments:**
  > **username_1**: which consul version you are using?  
  > **username_0**: Both servers and clients are on 1.0.0  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-20. Score: 4
  
  As a best practice, you should gracefully deregister the node gracefully. In this case, consul will know the the node has left and will be removed from the cluster. Otherwise, the consul cannot distinguish between a temporary failure, agent crash, network partition, etc.
  
  There is a github [issue](https://github.com/hashicorp/consul/issues/544) as well related to this.
  
  Quoting few important points from the issue.
  
  > The nodes should automatically reap out after 72hours (not yet configurable, but soon). Otherwise, the best route is to issue a graceful leave before destroying the nodes (consul leave), so that they can be reaped immediately. They are kept around for that long since without a graceful leave, Consul cannot distinguish between a temporary failure, agent crash, network partition, etc.
  
  .
  
  > force-leave should push them into the "left" state. Nodes are not reaped until they are in failed for 72h or in the left state. "force-leave" just moves a node from the "failed" -> "left" state. They are not removed from the members list for 24 or 72h.
  
  > **Comments:**
  > **username_0**: What I'm trying to figure out is that is there a configurable setting for reap time on the clients (for now only Consul servers have these settings). Gracefully leave is no longer sustainable for cloud environment anymore since VMs are very redundant and nobody cares whether a node is shutdown.  
  > **username_1**: As of now, this property is not available for agents. There is a github issue for enabling this. Please refer https://github.com/hashicorp/consul/issues/2982  
  > **username_2**: had to do the ugly force-leave command also to remove federation between 2 consul (VM - K8s) |  also doing the -prune will give me "the flag provided but not defined: -prune" | consul version 1.1.0  
  > **username_1**: @username_0 Would you be able to accept this answer if this had fixed the issue? This will help other people.  
  
  ---
  
  