title: couchbase Re-balance failing with error - Rebalance exited with reason {badmatch,failed}
body: <p>I am setting up a cluster. I tried to join 3 nodes but while re-balancing. I got below error. So i extracted some info from debug.log and unable to identify the exact issue. Appreciate any help.</p>
  
  <pre><code>=========================CRASH REPORT=========================
    crasher:
      initial call: service_agent:-spawn_connection_waiter/2-fun-0-/0
      pid: &lt;0.18486.7&gt;
      registered_name: []
      exception exit: {no_connection,"index-service_api"}
        in function  service_agent:wait_for_connection_loop/3 (src/service_agent.erl, line 305)
      ancestors: ['service_agent-index',service_agent_children_sup,
                    service_agent_sup,ns_server_sup,ns_server_nodes_sup,
                    &lt;0.170.0&gt;,ns_server_cluster_sup,&lt;0.89.0&gt;]
      messages: []
      links: [&lt;0.18481.7&gt;,&lt;0.18490.7&gt;]
      dictionary: []
      trap_exit: false
      status: running
      heap_size: 987
      stack_size: 27
      reductions: 1195
    neighbours:
  
  [ns_server:error,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:service_agent-index&lt;0.18481.7&gt;:service_agent:terminate:264]Terminating abnormally
  [ns_server:debug,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:&lt;0.18487.7&gt;:ns_pubsub:do_subscribe_link:145]Parent process of subscription {ns_config_events,&lt;0.18481.7&gt;} exited with reason {linked_process_died,
                                                                                    &lt;0.18486.7&gt;,
                                                                                    {no_connection,
                                                                                     "index-service_api"}}
  [error_logger:error,2018-02-12T13:54:43.531-05:00,ns_1@xuodf9.firebrand.com:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:do_log:203]** Generic server 'service_agent-index' terminating
  ** Last message in was {'EXIT',&lt;0.18486.7&gt;,
                                 {no_connection,"index-service_api"}}
  ** When Server state == {state,index,
                           {dict,6,16,16,8,80,48,
                            {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                            {{[[{uuid,&lt;&lt;"55a14ec6b06d72205b3cd956e6de60e7"&gt;&gt;}|
                                'ns_1@xuodf7.firebrand.com']],
                              [],
                              [[{uuid,&lt;&lt;"c5e67322a74826bef8edf27d51de3257"&gt;&gt;}|
                                'ns_1@xuodf8.firebrand.com']],
                              [],
                              [[{uuid,&lt;&lt;"3b55f7739e3fe85127dcf857a5819bdf"&gt;&gt;}|
                                'ns_1@xuodf9.firebrand.com']],
                              [],
                              [[{node,'ns_1@xuodf7.firebrand.com'}|
                                &lt;&lt;"55a14ec6b06d72205b3cd956e6de60e7"&gt;&gt;],
                               [{node,'ns_1@xuodf8.firebrand.com'}|
                                &lt;&lt;"c5e67322a74826bef8edf27d51de3257"&gt;&gt;],
                               [{node,'ns_1@xuodf9.firebrand.com'}|
                                &lt;&lt;"3b55f7739e3fe85127dcf857a5819bdf"&gt;&gt;]],
                              [],[],[],[],[],[],[],[],[]}}},
                           undefined,undefined,&lt;0.18626.7&gt;,#Ref&lt;0.0.5.56873&gt;,
                           &lt;0.18639.7&gt;,
                           {[{&lt;0.18646.7&gt;,#Ref&lt;0.0.5.56891&gt;}],[]},
                           undefined,undefined,undefined,undefined,undefined}
  ** Reason for termination ==
  ** {linked_process_died,&lt;0.18486.7&gt;,{no_connection,"index-service_api"}}
  
  [error_logger:error,2018-02-12T13:54:43.532-05:00,ns_1@xuodf9.firebrand.com:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:do_log:203]
  =========================CRASH REPORT=========================
    crasher:
      initial call: service_agent:init/1
      pid: &lt;0.18481.7&gt;
      registered_name: 'service_agent-index'
      exception exit: {linked_process_died,&lt;0.18486.7&gt;,
                          {no_connection,"index-service_api"}}
        in function  gen_server:terminate/6 (gen_server.erl, line 744)
      ancestors: [service_agent_children_sup,service_agent_sup,ns_server_sup,
                    ns_server_nodes_sup,&lt;0.170.0&gt;,ns_server_cluster_sup,
                    &lt;0.89.0&gt;]
      messages: [{'EXIT',&lt;0.18639.7&gt;,
                        {linked_process_died,&lt;0.18486.7&gt;,
                            {no_connection,"index-service_api"}}}]
      links: [&lt;0.18487.7&gt;,&lt;0.4805.0&gt;]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 28690
      stack_size: 27
      reductions: 6334
    neighbours:
  
  [error_logger:error,2018-02-12T13:54:43.533-05:00,ns_1@xuodf9.firebrand.com:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:do_log:203]
  =========================SUPERVISOR REPORT=========================
       Supervisor: {local,service_agent_children_sup}
       Context:    child_terminated
       Reason:     {linked_process_died,&lt;0.18486.7&gt;,
                                        {no_connection,"index-service_api"}}
       Offender:   [{pid,&lt;0.18481.7&gt;},
                    {name,{service_agent,index}},
                    {mfargs,{service_agent,start_link,[index]}},
                    {restart_type,permanent},
                    {shutdown,1000},
                    {child_type,worker}]
  
  
  [ns_server:error,2018-02-12T13:54:43.533-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index&lt;0.18626.7&gt;:service_rebalancer:run_rebalance:80]Agent terminated during the rebalance: {'DOWN',#Ref&lt;0.0.5.56860&gt;,process,
                                          &lt;0.18481.7&gt;,
                                          {linked_process_died,&lt;0.18486.7&gt;,
                                           {no_connection,"index-service_api"}}}
  [error_logger:info,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:do_log:203]
  =========================PROGRESS REPORT=========================
            supervisor: {local,service_agent_children_sup}
               started: [{pid,&lt;0.20369.7&gt;},
                         {name,{service_agent,index}},
                         {mfargs,{service_agent,start_link,[index]}},
                         {restart_type,permanent},
                         {shutdown,1000},
                         {child_type,worker}]
  
  [ns_server:error,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_agent-index&lt;0.20369.7&gt;:service_agent:handle_call:186]Got rebalance-only call {if_rebalance,&lt;0.18626.7&gt;,unset_rebalancer} that doesn't match rebalancer pid undefined
  [ns_server:error,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index&lt;0.18626.7&gt;:service_agent:process_bad_results:815]Service call unset_rebalancer (service index) failed on some nodes:
  [{'ns_1@xuodf9.firebrand.com',nack}]
  [ns_server:warn,2018-02-12T13:54:43.534-05:00,ns_1@xuodf9.firebrand.com:service_rebalancer-index&lt;0.18626.7&gt;:service_rebalancer:run_rebalance:89]Failed to unset rebalancer on some nodes:
  {error,{bad_nodes,index,unset_rebalancer,
                    [{'ns_1@xuodf9.firebrand.com',nack}]}}
  [error_logger:error,2018-02-12T13:54:43.535-05:00,ns_1@xuodf9.firebrand.com:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:do_log:203]
  =========================CRASH REPORT=========================
    crasher:
      initial call: service_rebalancer:-spawn_monitor/6-fun-0-/0
      pid: &lt;0.18626.7&gt;
      registered_name: 'service_rebalancer-index'
      exception exit: {linked_process_died,&lt;0.18486.7&gt;,
                          {no_connection,"index-service_api"}}
        in function  service_rebalancer:run_rebalance/7 (src/service_rebalancer.erl, line 92)
      ancestors: [cleanup_process,ns_janitor_server,ns_orchestrator_child_sup,
                    ns_orchestrator_sup,mb_master_sup,mb_master,&lt;0.4893.0&gt;,
                    ns_server_sup,ns_server_nodes_sup,&lt;0.170.0&gt;,
                    ns_server_cluster_sup,&lt;0.89.0&gt;]
      messages: [{'EXIT',&lt;0.18640.7&gt;,
                        {linked_process_died,&lt;0.18486.7&gt;,
                            {no_connection,"index-service_api"}}}]
      links: []
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 2586
      stack_size: 27
      reductions: 6359
    neighbours:
  
  [ns_server:error,2018-02-12T13:54:43.536-05:00,ns_1@xuodf9.firebrand.com:cleanup_process&lt;0.18625.7&gt;:service_janitor:maybe_init_topology_aware_service:84]Initial rebalance for `index` failed: {error,
                                         {initial_rebalance_failed,index,
                                          {linked_process_died,&lt;0.18486.7&gt;,
                                           {no_connection,
                                            "index-service_api"}}}}
  [ns_server:debug,2018-02-12T13:54:43.536-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth&lt;0.4796.0&gt;:menelaus_cbauth:handle_cast:95]Observed json rpc process {"projector-cbauth",&lt;0.5099.0&gt;} needs_update
  [ns_server:debug,2018-02-12T13:54:43.538-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth&lt;0.4796.0&gt;:menelaus_cbauth:handle_cast:95]Observed json rpc process {"goxdcr-cbauth",&lt;0.479.0&gt;} needs_update
  [ns_server:debug,2018-02-12T13:54:43.539-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth&lt;0.4796.0&gt;:menelaus_cbauth:handle_cast:95]Observed json rpc process {"cbq-engine-cbauth",&lt;0.5124.0&gt;} needs_update
  [ns_server:debug,2018-02-12T13:54:43.540-05:00,ns_1@xuodf9.firebrand.com:menelaus_cbauth&lt;0.4796.0&gt;:menelaus_cbauth:handle_cast:95]Observed json rpc process {"fts-cbauth",&lt;0.5129.0&gt;} needs_update
  </code></pre>
  
  <p>This is a blocker for cluster creation at this point.</p>