events.action: opened
events.author: bodgit
events.masked_author: username_0
events.text: I'm trying to attach a remote (potentially non-EC2) Tribe node to an EC2 cluster. I raised this on the mailing list but the thread died after a bit of back and forth so I figured I would raise this here as I would still like to get this fixed.
  
  I've created two nodes in EC2 EU region with the following configuration which is as small as possible to illustrate the problem:
  
  ```
  network.publish_host: "_ec2:publicDns_"
  discovery.type: ec2
  discovery.ec2.groups: estest
  discovery.ec2.host_type: public_dns
  cloud.aws.region: "eu-west-1"
  cloud.aws.access_key: abc123
  cloud.aws.secret_key: s3cr3t
  cloud.node.auto_attributes: true
  discovery.zen.minimum_master_nodes: 2
  discovery.zen.ping.multicast.enabled: false
  ```
  
  I'm using the public DNS host type because these resolve within EC2 to the private IP address but outside of EC2 return the public IP address which should theoretically mean an external node can join the cluster (the Tribe node in this case). Keeping traffic private within EC2 is meant to yield better performance avoiding the need to "hairpin" inter-cluster traffic as it would have to traverse the EC2 NAT layer if using the public IP addresses.
  
  Both nodes have ES 1.1.1 and cloud-aws 2.1.1 installed. Both are members of an ```estest``` security group which has the following rules:
  
  | Type | Protocol | Port Range | Source |
  | ----- | ----- | ----- | ----- |
  | Custom TCP Rule | TCP | 9300 - 9399 | sg-feedbeef (estest) |
  | Custom ICMP Rule | Echo Request | N/A | sg-feedbeef (estest) |
  | SSH | TCP | 22 | 1.2.3.4/32 |
  | Custom TCP Rule | TCP | 9200 | sg-feedbeef (estest) |
  | Custom TCP Rule | TCP | 9200 | 1.2.3.4/32 |
  | Custom TCP Rule |TCP | 9300 - 9399 | 1.2.3.4/32 |
  | Custom ICMP Rule | Echo Request | N/A | 1.2.3.4/32 |
  
  ```1.2.3.4``` is my IP address for external access. Both nodes have elastic IP addresses assigned to them so they are on "well known" external IP addresses for the purposes of the tribe node configuration.
  
  Both nodes correctly find each other on startup and I get a basic two-node cluster. Even though I'm using ```discovery.ec2.host_type: public_dns``` because the external DNS names internally resolve to the private IP addresses within EC2 the cluster is happily using the private IP addresses to communicate.
  
  However, if I query one node using ```http://54.72.215.117:9200/_nodes/transport?pretty``` I get:
  
  ```json
  {
    "cluster_name" : "elasticsearch",
    "nodes" : {
      "0nMUUSkXSnqy35ttLEkcqA" : {
        "name" : "Numinus",
        "transport_address" : "inet[/172.31.12.62:9300]",
        "host" : "ip-172-31-12-62",
        "ip" : "172.31.12.62",
        "version" : "1.1.1",
        "build" : "f1585f0",
        "http_address" : "inet[ec2-54-72-137-131.eu-west-1.compute.amazonaws.com/172.31.12.62:9200]",
        "attributes" : {
          "aws_availability_zone" : "eu-west-1a"
        },
        "transport" : {
          "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9300]",
          "publish_address" : "inet[/172.31.12.62:9300]"
        }
      },
      "-XlaNF-hSAi2U8tcpEFmug" : {
        "name" : "Howard the Duck",
        "transport_address" : "inet[ec2-54-72-215-117.eu-west-1.compute.amazonaws.com/172.31.12.61:9300]",
        "host" : "ip-172-31-12-61",
        "ip" : "172.31.12.61",
        "version" : "1.1.1",
        "build" : "f1585f0",
        "http_address" : "inet[ec2-54-72-215-117.eu-west-1.compute.amazonaws.com/172.31.12.61:9200]",
        "attributes" : {
          "aws_availability_zone" : "eu-west-1a"
        },
        "transport" : {
          "bound_address" : "inet[/0:0:0:0:0:0:0:0:9300]",
          "publish_address" : "inet[ec2-54-72-215-117.eu-west-1.compute.amazonaws.com/172.31.12.61:9300]"
        }
      }
    }
  }
  ```
  
  Notice that only the node that is the one I directly queried (Howard the Duck) has the public DNS included in its transport address. If I query the other node (Numinus) via ```http://54.72.137.131:9200/_nodes/transport?pretty``` I get:
  
  ```json
  {
    "cluster_name" : "elasticsearch",
    "nodes" : {
      "0nMUUSkXSnqy35ttLEkcqA" : {
        "name" : "Numinus",
        "transport_address" : "inet[ec2-54-72-137-131.eu-west-1.compute.amazonaws.com/172.31.12.62:9300]",
        "host" : "ip-172-31-12-62",
        "ip" : "172.31.12.62",
        "version" : "1.1.1",
        "build" : "f1585f0",
        "http_address" : "inet[ec2-54-72-137-131.eu-west-1.compute.amazonaws.com/172.31.12.62:9200]",
        "attributes" : {
          "aws_availability_zone" : "eu-west-1a"
        },
        "transport" : {
          "bound_address" : "inet[/0:0:0:0:0:0:0:0:9300]",
          "publish_address" : "inet[ec2-54-72-137-131.eu-west-1.compute.amazonaws.com/172.31.12.62:9300]"
        }
      },
      "-XlaNF-hSAi2U8tcpEFmug" : {
        "name" : "Howard the Duck",
        "transport_address" : "inet[/172.31.12.61:9300]",
        "host" : "ip-172-31-12-61",
        "ip" : "172.31.12.61",
        "version" : "1.1.1",
        "build" : "f1585f0",
        "http_address" : "inet[ec2-54-72-215-117.eu-west-1.compute.amazonaws.com/172.31.12.61:9200]",
        "attributes" : {
          "aws_availability_zone" : "eu-west-1a"
        },
        "transport" : {
          "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9300]",
          "publish_address" : "inet[/172.31.12.61:9300]"
        }
      }
    }
  }
  ```
  
  Notice that the other node (Numinus) now has the correct transport address. I would expect *both* nodes to have the same style transport address when queried from *either* node.
  
  My tribe node (also running ES 1.1.1) has the following minimal configuration:
  
  ```yaml
  discovery.zen.ping.multicast.enabled: false
  tribe:
    dublin:
      cluster:
        name: elasticsearch
      discovery:
        zen:
          ping:
            unicast:
              hosts:
                - 54.72.215.117
                - 54.72.137.131
  ```
  
  The tribe node correctly connects to the external IP address and (I'm guessing) retrieves the node list and then promptly tries to connect to the private IP addresses, and fails.
  
  Now I can change the EC2 cluster configuration to use the public IP addresses like so:
  
  ```
  network.publish_host: "_ec2:publicIp_"
  discovery.type: ec2
  discovery.ec2.groups: estest
  discovery.ec2.host_type: public_ip
  cloud.aws.region: "eu-west-1"
  cloud.aws.access_key: abc123
  cloud.aws.secret_key: s3cr3t
  cloud.node.auto_attributes: true
  discovery.zen.minimum_master_nodes: 2
  discovery.zen.ping.multicast.enabled: false
  ```
  
  However the cluster won't associate until I add the following additional rules to the ```estest``` security group:
  
  | Type | Protocol | Port Range | Source |
  | ----- | ----- | ----- | ----- |
  | Custom TCP Rule | TCP | 9300 - 9399 | 54.72.137.131/32 |
  | Custom TCP Rule | TCP | 9300 - 9399 | 54.72.215.117/32 |
  
  This isn't good as I can't easily scale the cluster without adding the public IP address of each node to the security group and also the performance is worse as every inter-cluster connection is hairpinning through the EC2 NAT layer.
  
  With the cluster up, I get consistent output from ```http://54.72.215.117:9200/_nodes/transport?pretty```, e.g.
  
  ```json
  {
    "cluster_name" : "elasticsearch",
    "nodes" : {
      "L4rjg5JNR1yuk-kU_eAthw" : {
        "name" : "Phil Urich",
        "transport_address" : "inet[/54.72.137.131:9300]",
        "host" : "ip-172-31-12-62",
        "ip" : "172.31.12.62",
        "version" : "1.1.1",
        "build" : "f1585f0",
        "http_address" : "inet[/54.72.137.131:9200]",
        "attributes" : {
          "aws_availability_zone" : "eu-west-1a"
        },
        "transport" : {
          "bound_address" : "inet[/0:0:0:0:0:0:0:0%0:9300]",
          "publish_address" : "inet[/54.72.137.131:9300]"
        }
      },
      "WvFPkVyfSi2dpHJ4k3yk8w" : {
        "name" : "Blue Streak",
        "transport_address" : "inet[/54.72.215.117:9300]",
        "host" : "ip-172-31-12-61",
        "ip" : "172.31.12.61",
        "version" : "1.1.1",
        "build" : "f1585f0",
        "http_address" : "inet[/54.72.215.117:9200]",
        "attributes" : {
          "aws_availability_zone" : "eu-west-1a"
        },
        "transport" : {
          "bound_address" : "inet[/0:0:0:0:0:0:0:0:9300]",
          "publish_address" : "inet[/54.72.215.117:9300]"
        }
      }
    }
  }
  ```
  
  I can then attach a tribe node successfully.
  
  There's a slight issue in that the tribe node must be either directly attached to the internet or have a 1:1 NAT with the external address set as the value for ```network.publish_host``` so that the EC2 cluster can talk back to the tribe node, but that's not directly related to this issue.
  
  It may be that a VPN is the only viable solution so you should just use private IP addresses everywhere and let the IPv4 routing layer deal with everything, or use IPv6, and then this issue goes away but the setting as it currently stands seems of little use.
events.title: Using "_ec2:publicDns_" for network.publish_host doesn't seem to work properly
events.type: issue
events.action: created
events.author: rparkhunovsky
events.masked_author: username_1
events.text: The issue is pretty actual for cross-region cloud environments and appears a blocker for usage as a feature.
events.type: comment
events.action: created
events.author: ssuprun
events.masked_author: username_2
events.text: I'm not sure if my change will fix this issue, but any way it looks like bug.
  Could somebody review it - https://github.com/elasticsearch/elasticsearch-cloud-aws/pull/175
events.type: comment
events.action: created
events.author: clintongormley
events.masked_author: username_3
events.text: @dadoonet with #175 merged, should this be closed? and maybe https://github.com/elastic/elasticsearch/issues/6333 too?
events.type: comment
events.action: created
events.author: rparkhunovsky
events.masked_author: username_1
events.text: Agree for closing #76. But tested the changes with elastic/elasticsearch#6333 - it doesn't resolve the issue anyway.
events.type: comment
events.action: created
events.author: clintongormley
events.masked_author: username_3
events.text: thanks @username_1 - closing this one
events.type: comment
events.action: created
events.author: taraslayshchuk
events.masked_author: username_4
events.text: Hi. I get in trouble with configurating 2 nodes in aws cloud in tribute cluster.
  
  Config file of first node:
  ```
  cluster.name: es_one
  node.name: node-t1
  
  plugin.mandatory: "cloud-aws"
  
  
  
  network.bound_address: "_ec2:privateDns_"
  network.public_host: "_ec2:publicDns_"
  
  
  transport.bind_host: "_ec2:privateDns_"
  transport.publish_host: "_ec2:publicDns_"
  
  discovery.zen.minimum_master_nodes: 1
  discovery.zen.ping.timeout: 10s
  discovery.zen.ping.multicast.enabled: false
  
  
  cloud:
       aws:
           access_key: key
           secret_key: value
  
  cloud.aws.region: "us-east-1"
  
  discovery.type: ec2
  ```
  Log output from first node:
  ```log
  [2015-09-02 15:06:18,828][INFO ][node                     ] [node-t1] version[1.7.1], pid[14686], build[b88f43f/2015-07-29T09:54:16Z]
  [2015-09-02 15:06:18,828][INFO ][node                     ] [node-t1] initializing ...
  [2015-09-02 15:06:18,925][INFO ][plugins                  ] [node-t1] loaded [cloud-aws], sites []
  [2015-09-02 15:06:18,987][INFO ][env                      ] [node-t1] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [5.3gb], net total_space [7.7gb], types [ext4]
  [2015-09-02 15:06:22,716][INFO ][node                     ] [node-t1] initialized
  [2015-09-02 15:06:22,716][INFO ][node                     ] [node-t1] starting ...
  [2015-09-02 15:06:22,790][INFO ][transport                ] [node-t1] bound_address {inet[/172.31.13.137:9301]}, publish_address {inet[ec2-52-3-85-143.compute-1.amazonaws.com/172.31.13.137:9301]}
  [2015-09-02 15:06:22,808][INFO ][discovery                ] [node-t1] es_one/qwcnD8D2QEest37d39onow
  [2015-09-02 15:06:33,992][INFO ][cluster.service          ] [node-t1] new_master [node-t1][qwcnD8D2QEest37d39onow][ip-172-31-13-137][inet[ec2-52-3-85-143.compute-1.amazonaws.com/172.31.13.137:9301]], reason: zen-disco-join (elected_as_master)
  [2015-09-02 15:06:34,021][INFO ][http                     ] [node-t1] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.31.13.137:9200]}
  [2015-09-02 15:06:34,022][INFO ][node                     ] [node-t1] started
  [2015-09-02 15:06:34,029][INFO ][gateway                  ] [node-t1] recovered [0] indices into cluster_state
  ```
  Config file of second node:
  ```
  cluster.name: es_two
  node.name: node-t2
  
  plugin.mandatory: "cloud-aws"
  
  network.host: "_ec2:publicDns_"
  
  network.bind_host: "_ec2:publicDns_"
  network.public_host: "_ec2:publicDns_"
  
  transport.host: "_ec2:publicDns_"
  transport.bind_host: "_ec2:publicDns_"
  transport.publish_host: "_ec2:publicDns_"
  
  discovery.zen.minimum_master_nodes: 1
  discovery.zen.ping.timeout: 10s
  discovery.zen.ping.multicast.enabled: false
  
  
  cloud:
       aws:
           access_key: key
           secret_key: value
  
  cloud.aws.region: "us-west-2"
  
  discovery.type: ec2
  ```
  Log output from second node:
  ```
  [2015-09-02 15:15:09,871][INFO ][node                     ] [node-t2] version[1.7.1], pid[2768], build[b88f43f/2015-07-29T09:54:16Z]
  [2015-09-02 15:15:09,872][INFO ][node                     ] [node-t2] initializing ...
  [2015-09-02 15:15:09,979][INFO ][plugins                  ] [node-t2] loaded [cloud-aws], sites []
  [2015-09-02 15:15:10,051][INFO ][env                      ] [node-t2] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [5.4gb], net total_space [7.7gb], types [ext4]
  [2015-09-02 15:15:13,801][INFO ][node                     ] [node-t2] initialized
  [2015-09-02 15:15:13,802][INFO ][node                     ] [node-t2] starting ...
  [2015-09-02 15:15:13,884][INFO ][transport                ] [node-t2] bound_address {inet[/192.168.1.155:9301]}, publish_address {inet[ec2-52-88-78-52.us-west-2.compute.amazonaws.com/192.168.1.155:9301]}
  [2015-09-02 15:15:13,908][INFO ][discovery                ] [node-t2] es_two/UIBV-NoFT8-YNPzIiqO8Jw
  [2015-09-02 15:15:25,380][INFO ][cluster.service          ] [node-t2] new_master [node-t2][UIBV-NoFT8-YNPzIiqO8Jw][ip-192-168-1-155][inet[ec2-52-88-78-52.us-west-2.compute.amazonaws.com/192.168.1.155:9301]], reason: zen-disco-join (elected_as_master)
  [2015-09-02 15:15:25,407][INFO ][http                     ] [node-t2] bound_address {inet[/192.168.1.155:9200]}, publish_address {inet[ec2-52-88-78-52.us-west-2.compute.amazonaws.com/192.168.1.155:9200]}
  [2015-09-02 15:15:25,407][INFO ][node                     ] [node-t2] started
  [2015-09-02 15:15:25,408][INFO ][gateway                  ] [node-t2] recovered [0] indices into cluster_state
  ```
  Local node config file:
  ```
  node.name: cluster
  
  
  plugin.mandatory: "cloud-aws"
  
  cloud:
       aws:
           access_key: key
           secret_key: value
  
  network.host: my.localhost.dns
  
  
  network.bind_host: my.localhost.dns
  
  network.public_host: my.localhost.dns
  
  
  transport.host: my.localhost.dns
  
  transport.bind_host: my.localhost.dns
  
  transport.publish_host: my.localhost.dns
  
  
  discovery.zen.minimum_master_nodes: 1
  discovery.zen.ping.timeout: 10s
  discovery.zen.ping.multicast.enabled: false
  
  tribe:
    es_one:
      cluster.name: es_one
      discovery.zen.ping.multicast.enabled: false
      discovery.type: ec2
      discovery.ec2.host_type: public_dns
      discovery.ec2.groups: sg-7b3c461c
  
  
    es_two:
      cluster.name: es_two
      discovery.zen.ping.multicast.enabled: false
      discovery.type: ec2
      discovery.ec2.host_type: public_dns
      discovery.ec2.groups: sg-8e55e0ea
  
  
  
  logger:
    level: DEBUG
  
  discovery.type: ec2
  ```
  Local node log file output:
  ```
  [2015-09-02 18:09:34,089][INFO ][node                     ] [cluster] version[1.7.1], pid[17251], build[b88f43f/2015-07-29T09:54:16Z]
  [2015-09-02 18:09:34,089][INFO ][node                     ] [cluster] initializing ...
  [2015-09-02 18:09:34,161][INFO ][plugins                  ] [cluster] loaded [cloud-aws], sites [bigdesk, head]
  [2015-09-02 18:09:35,998][INFO ][node                     ] [cluster/es_one] version[1.7.1], pid[17251], build[b88f43f/2015-07-29T09:54:16Z]
  [2015-09-02 18:09:35,998][INFO ][node                     ] [cluster/es_one] initializing ...
  [2015-09-02 18:09:35,998][INFO ][plugins                  ] [cluster/es_one] loaded [cloud-aws], sites []
  [2015-09-02 18:09:37,157][INFO ][node                     ] [cluster/es_one] initialized
  [2015-09-02 18:09:37,159][INFO ][node                     ] [cluster/es_two] version[1.7.1], pid[17251], build[b88f43f/2015-07-29T09:54:16Z]
  [2015-09-02 18:09:37,159][INFO ][node                     ] [cluster/es_two] initializing ...
  [2015-09-02 18:09:37,160][INFO ][plugins                  ] [cluster/es_two] loaded [cloud-aws], sites []
  [2015-09-02 18:09:37,845][INFO ][node                     ] [cluster/es_two] initialized
  [2015-09-02 18:09:37,857][INFO ][node                     ] [cluster] initialized
  [2015-09-02 18:09:37,857][INFO ][node                     ] [cluster] starting ...
  [2015-09-02 18:09:37,919][INFO ][transport                ] [cluster] bound_address {inet[/10.131.72.169:9301]}, publish_address {inet[tlaispc.ddns.softservecom.com/10.131.72.169:9301]}
  [2015-09-02 18:09:37,925][INFO ][discovery                ] [cluster] elasticsearch/xFarBkexRqKrBw6gezgPdA
  [2015-09-02 18:09:37,925][WARN ][discovery                ] [cluster] waited for 0s and no initial state was set by the discovery
  [2015-09-02 18:09:37,929][INFO ][http                     ] [cluster] bound_address {inet[/10.131.72.169:9200]}, publish_address {inet[tlaispc.ddns.softservecom.com/10.131.72.169:9200]}
  [2015-09-02 18:09:37,929][INFO ][node                     ] [cluster/es_one] starting ...
  [2015-09-02 18:09:37,938][INFO ][transport                ] [cluster/es_one] bound_address {inet[/0:0:0:0:0:0:0:0:9302]}, publish_address {inet[/10.131.72.169:9302]}
  [2015-09-02 18:09:37,942][INFO ][discovery                ] [cluster/es_one] es_one/DohOLLs9QwqCk4eKt9VDFw
  [2015-09-02 18:10:07,942][WARN ][discovery                ] [cluster/es_one] waited for 30s and no initial state was set by the discovery
  [2015-09-02 18:10:07,943][INFO ][node                     ] [cluster/es_one] started
  [2015-09-02 18:10:07,943][INFO ][node                     ] [cluster/es_two] starting ...
  [2015-09-02 18:10:07,951][INFO ][transport                ] [cluster/es_two] bound_address {inet[/0:0:0:0:0:0:0:0:9303]}, publish_address {inet[/10.131.72.169:9303]}
  [2015-09-02 18:10:07,953][INFO ][discovery                ] [cluster/es_two] es_two/IJVyDR_lRuOoE3bqlSnpGQ
  [2015-09-02 18:10:37,953][WARN ][discovery                ] [cluster/es_two] waited for 30s and no initial state was set by the discovery
  [2015-09-02 18:10:37,954][INFO ][node                     ] [cluster/es_two] started
  [2015-09-02 18:10:37,954][INFO ][node                     ] [cluster] started
  ```
  So what we have? Nodes see each other, but did not make cluster. I don`t know why, but cluster.service did not start before discovery. Can you help me please - what I am doing wrong?
events.type: comment
events.action: created
events.author: Pryz
events.masked_author: username_5
events.text: I'm also hitting this bug. Any news on this ?
  
  elasticsearch-cloud-aws : 2.7.1
  elasticsearch :  1.7.1
events.type: comment
events.action: created
events.author: toleksyn
events.masked_author: username_6
events.text: Also encountered this one with es 1.6. Please fix.
events.type: comment
events.action: created
events.author: RaffaelloBertini
events.masked_author: username_7
events.text: @username_4  you have different cluster name among the nodes, that way they didn't build a cluster.
events.type: comment
events.action: created
events.author: Raffaello
events.masked_author: username_8
events.text: @username_4 you have different cluster name among the nodes, that way they didn't build a cluster.
events.type: comment
events.action: closed
events.author: bodgit
events.masked_author: username_0
events.type: issue