title: Spring Boot Admin with Docker Swarm Unable to pull Health Metrics
body: <p>I am trying to get spring boot admin to work in a docker swarm cluster using a zookeeper discovery service mechanism to ensure that all clients are dynamically discovered once connected to zookeeper.  Problem is it appears springboot admin is unable to reach the health actuator endpoints on the clients due to a connection refused even though all docker services are using the same overlay network and each container can ping one another which ive verified via docker exec -it  ping  to ensure they are all reachable from one another.</p>
  
  <p>Ive also verified that the clients and the admin service's are properly connecting to zookeeper and that zookeeper + admin dashboard are infact seeing those clients as registered. </p>
  
  <p>To recreate this issue ive created a simple docker compose that deploys two spring boot admin apps with the actuators enabled over the same overlay network via the compose file below:</p>
  
  <pre><code>version: '3.1'
  
  services:
      zoo1:
          image: zookeeper:3.4.12
          hostname: zoo1
          networks:
              - nsp_test
          deploy:
              restart_policy:
                  condition: on-failure
              placement:
                  constraints: [node.hostname == nj51nreda5v]
          environment:
              ZOO_MY_ID: 1
              ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888
  
      zoo2:
          image: zookeeper:3.4.12
          hostname: zoo2
          networks:
              - nsp_test
          deploy:
              restart_policy:
                  condition: on-failure
              placement:
                  constraints: [node.hostname == nj51nreda6v]
          environment:
              ZOO_MY_ID: 2
              ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888
  
      nspadmin:
          image: admin:77
          ports:
              - "9084:8080"
          networks:
              - nsp_test
          depends_on:
              - "zoo1"
              - "zoo2"
          deploy:
              restart_policy:
                  condition: on-failure
              mode: global
          environment:
              ZK_HOST: zoo1:2181,zoo2:2182
              SPRING_PROFILES_ACTIVE: ssldev
  networks:
      nsp_test:
        external:
          name: nsp_test
  </code></pre>
  
  <p>From this configuration I see both spring admin dashboards registered in zookeeper and display as OFFLINE (since it cant reach the /health actuator)</p>
  
  <p>The following two addresses are what it registers for the clients in SBA.
  <a href="https://10.255.0.19:8080/" rel="nofollow noreferrer">https://10.255.0.19:8080/</a> OFFLINE
  <a href="https://10.255.0.20:8080/" rel="nofollow noreferrer">https://10.255.0.20:8080/</a> OFFLINE</p>
  
  <p>The exception I get.</p>
  
  <pre><code>2018-12-31 04:20:31.926  INFO 1 --- [    updateTask1] d.c.boot.admin.registry.StatusUpdater    : Couldn't retrieve status for Application [id=28eab1e1, name=nsp-admin, managementUrl=https://10.255.0.20:8080/, healthUrl=https://10.255.0.20:8080/health, serviceUrl=https://10.255.0.20:8080/]
  org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://10.255.0.20:8080/health": Connect to 10.255.0.20:8080 [/10.255.0.20] failed: connect timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to 10.255.0.20:8080 [/10.255.0.20] failed: connect timed out
          at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:549) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          at de.codecentric.boot.admin.web.client.ApplicationOperations.doGet(ApplicationOperations.java:68) ~[spring-boot-admin-server-1.5.6.jar!/:1.5.6]
          at de.codecentric.boot.admin.web.client.ApplicationOperations.getHealth(ApplicationOperations.java:58) ~[spring-boot-admin-server-1.5.6.jar!/:1.5.6]
          at de.codecentric.boot.admin.registry.StatusUpdater.queryStatus(StatusUpdater.java:111) [spring-boot-admin-server-1.5.6.jar!/:1.5.6]
          at de.codecentric.boot.admin.registry.StatusUpdater.updateStatus(StatusUpdater.java:65) [spring-boot-admin-server-1.5.6.jar!/:1.5.6]
          at de.codecentric.boot.admin.registry.StatusUpdateApplicationListener$1.run(StatusUpdateApplicationListener.java:47) [spring-boot-admin-server-1.5.6.jar!/:1.5.6]
          at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_151]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_151]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
          at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
  Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 10.255.0.20:8080 [/10.255.0.20] failed: connect timed out
          at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:89) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652) ~[spring-web-4.3.8.RELEASE.jar!/:4.3.8.RELEASE]
          ... 15 common frames omitted
  Caused by: java.net.SocketTimeoutException: connect timed out
          at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_151]
          at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_151]
          at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_151]
          at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_151]
          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_151]
          at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_151]
          at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) ~[httpclient-4.5.3.jar!/:4.5.3]
          at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.3.jar!/:4.5.3]
  </code></pre>
  
  <p>My SBA Configuration yml</p>
  
  <pre><code>server:
    port: 8080
  spring:
    boot:
      admin:
        client:
          prefer-ip: false
    datasource:
      driverClassName: org.postgresql.Driver
      url: ${DB_URL}
      username: ${DB_USER}
      password: ${DB_PASSWORD}
    application:
      name: nsp-admin
    cloud:
      config:
        discovery:
          enabled: true
      zookeeper:
        connect-string: ${ZK_HOST}
        discovery:
          uri-spec: https://{address}:{port}
          metadata:
            management:
              context-path: /
            health:
              path: /health
  
  management:
    security:
      enabled: false
  
  security:
    basic:
      enabled: false
  
  #security.require-ssl: true
  server.ssl.enabled: true
  server.ssl.key-store-type: PKCS12
  server.ssl.key-store: *****
  server.ssl.key-store-password: *****
  </code></pre>
  
  <p><strong>UPDATE</strong>
  After debugging the issue a bit more I realized it without a doubt has to do with the hostname/IP the clients are registering in zookeeper. </p>
  
  <p>When i perform a curl using the docker id's as the hostnames the /health api returns when performing the curl from SBA to the Client container id. </p>
  
  <p>This Works:
  docker exec -it 8403c5001b9e curl -k <a href="https://bf41c73af594:8080/health" rel="nofollow noreferrer">https://bf41c73af594:8080/health</a></p>
  
  <p>This does not work results in a timeout: docker exec -it 8403c5001b9e curl -k <a href="https://10.255.0.20:8080/health" rel="nofollow noreferrer">https://10.255.0.20:8080/health</a></p>
  
  <p>Is it possible to force zookeeper to register the hostname or the containerid instead?  </p>
  
  <p><strong>UPDATE</strong>
  Setting spring.cloud.zookeeper.discovery.instanceHost: ${HOSTNAME} in my application.yml resolves the issue.  It forces the correct containerId to be registered to zookeeper.  </p>