title: Neo4j cluster not starting
body: <p>I am building a Neo4j HA cluster and have followed the steps at <a href="https://neo4j.com/docs/operations-manual/current/tutorial/highly-available-cluster/" rel="nofollow noreferrer">https://neo4j.com/docs/operations-manual/current/tutorial/highly-available-cluster/</a></p>
  
  <p>I have 3 machines running Neo4j in a single-instance Docker container.</p>
  
  <p>Neo4j works fine as a standalone server on each machine, but when I add the HA config, it fails to start.</p>
  
  <p>Each machine was created from the same prebuilt image so they are configured identically except for the ha.server_id which is unique for each machine (values are 1, 2, 3).</p>
  
  <p>Firewall is off and I've verified connectivity between the machines.</p>
  
  <p>The Neo4j HA config is:</p>
  
  <pre><code>ha.server_id=1
  ha.initial_hosts=172.0.30.110:5001,172.0.31.39:5001,172.0.32.249:5001
  dbms.mode=HA
  </code></pre>
  
  <p>To start Neo4j I'm running the following on each machine:</p>
  
  <pre><code>/usr/bin/docker run \
  --publish=7474:7474 --publish=7687:7687 --publish=5001:5001 \
  --volume=/var/lib/neo4j/data:/data \
  --volume=/var/lib/neo4j/logs:/logs \
  --volume=/var/lib/neo4j/conf:/conf \
  --name=neo4j \
  neo4j:3.0-enterprise
  </code></pre>
  
  <p>When I run the startup command on machines 2 and 3, and sometimes on machine 1, I get an error following 'Attempting to join cluster'.  Occasionally (but not all the time) when I run the command on machine 1, it realizes it is the first machine and goes ahead to initialize itself as master.</p>
  
  <p>Below are the full failure and success messages.  I can't figure out why only one machine (incidentally the first) is doing what it's supposed to do sometimes and not all the time.  I also need to figure out how to get the other two machines to register themselves with the cluster.</p>
  
  <ol>
  <li>Why is one node sometimes launching Neo4j with clustering and sometimes not?</li>
  <li>Why are the 2nd and 3rd nodes not registering with the cluster?</li>
  <li>Where should I look next?</li>
  </ol>
  
  <p>Error (seen on all 3 machines):</p>
  
  <pre><code>&gt; 2017-05-30 09:18:35.502+0000 INFO  Starting... 2017-05-30
  &gt; 09:18:35.833+0000 INFO  Write transactions to database disabled
  &gt; 2017-05-30 09:18:36.116+0000 INFO  Bolt enabled on 0.0.0.0:7687.
  &gt; 2017-05-30 09:18:36.131+0000 INFO  Initiating metrics... 2017-05-30
  &gt; 09:18:36.937+0000 INFO  Attempting to join cluster of
  &gt; [172.0.30.110:5001, 172.0.31.39:5001, 172.0.32.249:5001] 2017-05-30
  &gt; 09:19:06.996+0000 ERROR Failed to start Neo4j: Starting Neo4j failed:
  &gt; Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception. Starting Neo4j failed: Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception. org.neo4j.server.ServerStartupException: Starting
  &gt; Neo4j failed: Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception.
  &gt;         at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
  &gt;         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:215)
  &gt;         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:90)
  &gt;         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:67)
  &gt;         at org.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:32)
  &gt; Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception.
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
  &gt;         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
  &gt;         ... 3 more Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory,
  &gt; /var/lib/neo4j/data/databases/fraud.db
  &gt;         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
  &gt;         at org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory.newFacade(HighlyAvailableFacadeFactory.java:42)
  &gt;         at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.&lt;init&gt;(HighlyAvailableGraphDatabase.java:41)
  &gt;         at org.neo4j.server.enterprise.EnterpriseNeoServer.lambda$static$0(EnterpriseNeoServer.java:80)
  &gt;         at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
  &gt;         ... 5 more Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
  &gt; 'org.neo4j.cluster.client.ClusterJoin@5d2e6054' was successfully
  &gt; initialized, but failed to start. Please see attached cause exception.
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
  &gt;         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
  &gt;         ... 10 more Caused by: java.util.concurrent.TimeoutException: Conversation-response mapping:
  &gt; {3/13#=ResponseFuture{conversationId='3/13#',
  &gt; initiatedByMessageType=join, response=null}}
  &gt;         at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:314)
  &gt;         at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:143)
  &gt;         at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:82)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
  &gt;         ... 14 more ubuntu@ip-172-0-32-249:~$ sudo /usr/bin/docker rm neo4j neo4j ubuntu@ip-172-0-32-249:~$ sudo /usr/bin/docker run
  &gt; --publish=7474:7474 --publish=7687:7687 --publish=5001:5001 --volume=/var/lib/neo4j/data:/data --volume=/var/lib/neo4j/logs:/logs --volume=/var/lib/neo4j/conf:/conf --name=neo4j neo4j:3.0-enterprise Starting Neo4j. 2017-05-30 09:20:17.877+0000 INFO  No SSL certificate
  &gt; found, generating a self-signed certificate.. 2017-05-30
  &gt; 09:20:18.387+0000 INFO  Starting... 2017-05-30 09:20:18.676+0000 INFO 
  &gt; Write transactions to database disabled 2017-05-30 09:20:19.036+0000
  &gt; INFO  Bolt enabled on 0.0.0.0:7687. 2017-05-30 09:20:19.046+0000 INFO 
  &gt; Initiating metrics... 2017-05-30 09:20:19.779+0000 INFO  Attempting to
  &gt; join cluster of [172.0.30.110:5001, 172.0.31.39:5001,
  &gt; 172.0.32.249:5001] 2017-05-30 09:20:49.851+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception. Starting Neo4j failed: Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception. org.neo4j.server.ServerStartupException: Starting
  &gt; Neo4j failed: Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception.
  &gt;         at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
  &gt;         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:215)
  &gt;         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:90)
  &gt;         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:67)
  &gt;         at org.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:32)
  &gt; Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
  &gt; 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
  &gt; successfully initialized, but failed to start. Please see attached
  &gt; cause exception.
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
  &gt;         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
  &gt;         ... 3 more Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory,
  &gt; /var/lib/neo4j/data/databases/fraud.db
  &gt;         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
  &gt;         at org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory.newFacade(HighlyAvailableFacadeFactory.java:42)
  &gt;         at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.&lt;init&gt;(HighlyAvailableGraphDatabase.java:41)
  &gt;         at org.neo4j.server.enterprise.EnterpriseNeoServer.lambda$static$0(EnterpriseNeoServer.java:80)
  &gt;         at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
  &gt;         ... 5 more Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
  &gt; 'org.neo4j.cluster.client.ClusterJoin@6c1c89e2' was successfully
  &gt; initialized, but failed to start. Please see attached cause exception.
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
  &gt;         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
  &gt;         ... 10 more Caused by: java.util.concurrent.TimeoutException: Conversation-response mapping:
  &gt; {3/13#=ResponseFuture{conversationId='3/13#',
  &gt; initiatedByMessageType=join, response=null}}
  &gt;         at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:314)
  &gt;         at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:143)
  &gt;         at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:82)
  &gt;         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
  &gt;         ... 14 more
  </code></pre>
  
  <p>Success (seen on one machine only):</p>
  
  <pre><code>2017-05-30 09:20:22.963+0000 INFO  Starting...
  2017-05-30 09:20:23.319+0000 INFO  Write transactions to database disabled
  2017-05-30 09:20:23.619+0000 INFO  Bolt enabled on 0.0.0.0:7687.
  2017-05-30 09:20:23.628+0000 INFO  Initiating metrics...
  2017-05-30 09:20:24.352+0000 INFO  Attempting to join cluster of [172.0.30.110:5001, 172.0.31.39:5001, 172.0.32.249:5001]
  2017-05-30 09:20:31.382+0000 INFO  Could not join cluster of [172.0.30.110:5001,172.0.31.39:5001, 172.0.32.249:5001]
  2017-05-30 09:20:31.382+0000 INFO  Creating new cluster with name [neo4j.ha]...
  2017-05-30 09:20:31.396+0000 INFO  Instance 1 (this server)  entered the cluster
  2017-05-30 09:20:31.401+0000 INFO  Instance 1 (this server)  was elected as coordinator
  2017-05-30 09:20:31.413+0000 INFO  I am 1, moving to master
  2017-05-30 09:20:31.465+0000 INFO  Instance 1 (this server)  was elected as coordinator
  2017-05-30 09:20:31.492+0000 INFO  I am 1, successfully moved to master
  2017-05-30 09:20:31.510+0000 INFO  Instance 1 (this server)  is available as master at ha://172.17.0.2:6001?serverId=1 with StoreId{creationTime=1496122128110,randomId=-2708986986476371425, storeVersion=15531981201765894, upgradeTime=1496122128110, upgradeId=1}
  2017-05-30 09:20:31.612+0000 INFO  Instance 1 (this server)  is available as backup at backup://127.0.0.1:6362 with StoreId{creationTime=1496122128110, randomId=-2708986986476371425, storeVersion=15531981201765894, upgradeTime=1496122128110, upgradeId=1}
  2017-05-30 09:20:31.709+0000 INFO  Database available for write transactions
  2017-05-30 09:20:32.929+0000 INFO  Started.
  2017-05-30 09:20:33.087+0000 INFO  Mounted REST API at: /db/manage
  2017-05-30 09:20:33.828+0000 INFO  Remote interface available at http://0.0.0.0:7474/
  </code></pre>