Answers: <p>This might not be the answer to your specific cross data-center-ip setup.</p>
  
  <p>I occasionally run into one or more swarm-nodes being status: <code>Down</code> and Availability: <code>Active</code>. Having the  <code>Status.Message</code>: <code>"heartbeat failure"</code>. This can happen after a reboot.</p>
  
  <p>What helped was to stop the docker daemon, remove <code>/var/lib/docker/swarm/worker/tasks.db</code> and start the docker daemon again.</p>
  
  <p>from:
  <a href="https://github.com/moby/moby/issues/34827#issuecomment-457678500" rel="nofollow noreferrer">https://github.com/moby/moby/issues/34827#issuecomment-457678500</a></p>
  
  <p>sometimes it fixes itself: <a href="https://stackoverflow.com/a/54126180/2087704">https://stackoverflow.com/a/54126180/2087704</a></p>
  
Answers: CC BY-SA 4.0
Answers: 2020-04-08T07:34:20.440Z
Answers: 2020-04-08T07:34:20.440Z
Answers: invad0r
Body: <p>My cluster is currently located in a single data center. I've been trying to change that by adding a single worker node from another data center, but so far it hasn't worked.</p>
  
  <p>I'm able to make this node join the swarm and get listed by the managers, but it is always shown as "Down". Here is what "docker inspect" shows me about this node:</p>
  
  <pre><code>    "Status": {
          "State": "down",
          "Message": "heartbeat failure",
          "Addr": "xxx.xxx.xxx.xxx"
      }
  </code></pre>
  
  <p>I've opened the following ports in both sides:</p>
  
  <pre><code>2377 tcp
  7946 tcp+udp
  4789 udp
  </code></pre>
  
  <p>How do I troubleshoot and fix this?</p>
  
Comments: CC BY-SA 4.0
Comments: 2018-10-09T02:25:05.723Z
Comments: Found it. My nodes are advertising their private IPs to the external node. Not sure how to fix that, but I guess it's another question
Comments: ivarec
Comments: CC BY-SA 4.0
Comments: 2019-12-23T09:22:04.310Z
Comments: Could you post an answer?
Comments: 030
Tags: <docker><docker-swarm><networking>
Title: How to fix a "heartbeat failure" in Docker Swarm?
UserDisplayName: ivarec
ThreadText: # How to fix a "heartbeat failure" in Docker Swarm?
  
  (Asked by: username_0 on 2018-10-09)
  
  My cluster is currently located in a single data center. I've been trying to change that by adding a single worker node from another data center, but so far it hasn't worked.
  
  I'm able to make this node join the swarm and get listed by the managers, but it is always shown as "Down". Here is what "docker inspect" shows me about this node:
  
  ```
      "Status": {
          "State": "down",
          "Message": "heartbeat failure",
          "Addr": "xxx.xxx.xxx.xxx"
      }
  
  ```
  
  I've opened the following ports in both sides:
  
  ```
  2377 tcp
  7946 tcp+udp
  4789 udp
  
  ```
  
  How do I troubleshoot and fix this?
  
  **Tags:** docker, docker-swarm, networking
  
  > **Comments:**
  > **username_0**: Found it. My nodes are advertising their private IPs to the external node. Not sure how to fix that, but I guess it's another question  
  > **username_1**: Could you post an answer?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-08. Score: 2
  
  This might not be the answer to your specific cross data-center-ip setup.
  
  I occasionally run into one or more swarm-nodes being status: `Down` and Availability: `Active`. Having the `Status.Message`: `"heartbeat failure"`. This can happen after a reboot.
  
  What helped was to stop the docker daemon, remove `/var/lib/docker/swarm/worker/tasks.db` and start the docker daemon again.
  
  from:
  <https://github.com/moby/moby/issues/34827#issuecomment-457678500>
  
  sometimes it fixes itself: <https://stackoverflow.com/a/54126180/2087704>
  
  
  ---
  
  