Answers: <p>With help from a Microsoft Engineer, we were able to resolve our issue.</p>
  
  <p>The trick to it was that the node that could communicate with the outside world was always the one that the kubernetes tunnel pod was running on.  This indicated a problem with communication between nodes.  The traceroute indicated that this, with no network security group traffic restrictions, was bouncing between the node1 ip, to the load balancer ip, back to the node2 ip.  </p>
  
  <p>we had a security rule that restricted all traffic that was not on our specifically approved list.</p>
  
  <p>One of the debug steps I had taken was to open all traffic between the load balancer ip to the load balancer ip.  However, this failed to consider the fact that the traffic was not from the load balancer to the load balancer, but from nodex to the load balancer then from the load balancer to nodey.  </p>
  
  <p>We added an inbound and outbound traffic rule that allowed traffic from the subnet ips that nodes can possibly be on to the load balancer, as well as within the same subnet for good meansure.  This resolved the issue.</p>
  
  <p>If that's confusing, I can provide images that show the new rules and explain it a little clearer, but I'm hoping that is clear enough for anyone who experiences a similar issue.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-11-12T20:13:50.887Z
Answers: 2018-11-12T20:13:50.887Z
Answers: Marshall Tigerus
Answers: <p>What about creating a new managed k8s cluster and start from scratch? This is what I do if the architecture is a maze. Managed k8s on azure and gcp is a couple of clicks away.</p>
  
Answers: CC BY-SA 4.0
Answers: 2018-10-30T16:37:44.017Z
Answers: We have a support ticket oustanding to microsoft, but if they don't get back to us soon, I'm going to try this.
Answers: Marshall Tigerus
Answers: CC BY-SA 4.0
Answers: 2018-10-30T22:15:15.310Z
Answers: You could already start. You could leave the current cluster running and create a new managed cluster next to it without destroying the existing one.
Answers: 030
Answers: CC BY-SA 4.0
Answers: 2018-10-31T18:25:16.290Z
Answers: more of a matter of time than anything else.  I have a lot of other responsibilities to take care of, and technically this is microsoft's problem at the moment.
Answers: Marshall Tigerus
Answers: CC BY-SA 4.0
Answers: 2018-10-29T21:36:09.680Z
Answers: 2018-10-29T21:36:09.680Z
Answers: 030
Answers: <p>We had the same issue; pods on one node couldn't communicate with pods on another.</p>
  <h2>Testing Connectivity</h2>
  <p>We first confirmed our theory that pods couldn't communicate by using <code>kubectl exec -it &lt;podname&gt; -- /bin/bash</code> to connect to a pod on one node (<a href="https://kubernetes.io/docs/tasks/debug/debug-application/get-shell-running-container/#getting-a-shell-to-a-container" rel="nofollow noreferrer">related docs</a>).
  From that pod we then tried to connect to open ports on pods on the same node and on different nodes; thus showing that we could communicate with those on the same node, but not with those on a different node / confirming our theory.</p>
  <p>Since most pods don't have utility tooling installed, we used the <code>(timeout 1 bash -c '&lt;/dev/tcp/10.244.1.45/8089' &amp;&amp; echo PORT OPEN || echo PORT CLOSED) 2&gt;/dev/null</code> approach to test connectivity (where <code>10.244.1.45</code> is the IP of the target pod, and 8089 the listening port).  <a href="https://stackoverflow.com/a/19866239/361842">Related notes</a></p>
  <h2>Resolving the issue</h2>
  <p>In our case the issue was with the Route Table associated with our Kubernetes cluster's subnet.</p>
  <ul>
  <li>Open the Azure Portal</li>
  <li>Navigate to the kubernetes service</li>
  <li>Navigate: <code>Settings</code> &gt; <code>Networking</code> &gt; <code>Virtual Network Integration</code></li>
  <li>Click on the subnet</li>
  <li>Click on the route table associated with our subnet</li>
  <li>Under routes you'll see each route is named after the node to which it relates.  The <code>Next hop IP address</code> is the IP of that node.  The <code>Address prefix</code> is the CIDR under which all pods hosted on that node should sit.</li>
  <li><strong>This was the issue:</strong> we found that the <code>address prefix</code> associated with one node was different to the pods addresses on that node; i.e. the address prefix was <code>10.244.0.0/24</code>, but pods under that node were in the range <code>10.244.1.0/24</code>; e.g. <code>10.244.1.45</code>.</li>
  <li>We resolved the issue by adding a new route with the correct information (we gave it the same name as the invalid route, just suffixing <code>_fix</code> on the end - we could probably have just corrected the invalid entry, but adding a new route felt lower risk).</li>
  </ul>
  <p>Hope that helps others who may hit this issue.</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-07-02T13:53:09.000Z
Answers: 2024-07-02T13:53:09.000Z
Answers: JohnLBevan
Body: <p>I'll start off by saying I'm not really an infrastructure person, but am being required to maintain it until we hire someone more suited to it, so apologies if I have any terminology wrong.</p>
  
  <p>We have a kubernetes cluster (AKS)in azure.  Currently we are running only 1 server in the cluster, with 4 namespaces configured within it, representing each environment.  To properly support our production environment, we ideally want at least 3 servers, with the ability to scale appropriately.  However, whenever we try to scale it up, services deployed to different server nodes cannot communicate with one another.</p>
  
  <p>The unfortunate thing is I have no idea even how to start begin debugging this.  I know we have some custom nginx rules that microsoft gave us, but even setting allow everything as the first rule does not seem to resolve this issue.  How do I begin debugging this?  What information would even be helpful for this issue?</p>
  
Comments: CC BY-SA 4.0
Comments: 2018-10-26T18:55:21.047Z
Comments: I'm not very knowledgeable with azure, but I think each node has a kind of firewall on azure level, like security groups on aws, and you need to open the inbound ports for k8s, depending on you k8s setup there's too much options to give more guidance without details on the cluster configuration
Comments: Tensibai
Comments: CC BY-SA 4.0
Comments: 2018-10-26T21:40:32.697Z
Comments: 4 namespaces? How many services have been deployed? What does kubectl get nodes and get ns return? Please provide some more information regarding the topology and k8s version. It is not clear to me whether you use the managed k8s or whether you deployed it yourself. Are you sure that all svc run? What apps should communicate with each other?
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2018-10-29T17:41:42.697Z
Comments: @030 kubectl get nodes returns 1 node currently.  It returns 3 when we have the scaling turned up to 3 on the azure portal.  get ns returns 9 namespaces, our 4 environments (dev, test, uat, prod), azure-system, default, kube-public, kube-system, and tiller
Comments: Marshall Tigerus
Comments: CC BY-SA 4.0
Comments: 2018-10-29T17:42:42.177Z
Comments: @030 additionally, we hired a contractor that set up this environment, so I'm not sure everything he did.  He provided us documentation that includes kubectl commands that were run to set things up, but I think he was just following advice from microsoft and things he googled, as he didn't seem to have any expertise in kubernetes.
Comments: Marshall Tigerus
Comments: CC BY-SA 4.0
Comments: 2018-10-29T17:43:27.143Z
Comments: @Tensibai the firewall settings seemed to be at a higher level, but I've been messing with them to try to get things working.
Comments: Marshall Tigerus
Tags: <kubernetes><azure><azure-kubernetes-service>
Title: nodes in a kubernetes cluster in azure unable to communicate with one another
UserDisplayName: Marshall Tigerus
ThreadText: # nodes in a kubernetes cluster in azure unable to communicate with one another
  
  (Asked by: username_0 on 2018-10-26)
  
  I'll start off by saying I'm not really an infrastructure person, but am being required to maintain it until we hire someone more suited to it, so apologies if I have any terminology wrong.
  
  We have a kubernetes cluster (AKS)in azure. Currently we are running only 1 server in the cluster, with 4 namespaces configured within it, representing each environment. To properly support our production environment, we ideally want at least 3 servers, with the ability to scale appropriately. However, whenever we try to scale it up, services deployed to different server nodes cannot communicate with one another.
  
  The unfortunate thing is I have no idea even how to start begin debugging this. I know we have some custom nginx rules that microsoft gave us, but even setting allow everything as the first rule does not seem to resolve this issue. How do I begin debugging this? What information would even be helpful for this issue?
  
  **Tags:** kubernetes, azure, azure-kubernetes-service
  
  > **Comments:**
  > **username_1**: I'm not very knowledgeable with azure, but I think each node has a kind of firewall on azure level, like security groups on aws, and you need to open the inbound ports for k8s, depending on you k8s setup there's too much options to give more guidance without details on the cluster configuration  
  > **username_2**: 4 namespaces? How many services have been deployed? What does kubectl get nodes and get ns return? Please provide some more information regarding the topology and k8s version. It is not clear to me whether you use the managed k8s or whether you deployed it yourself. Are you sure that all svc run? What apps should communicate with each other?  
  > **username_0**: @username_2 kubectl get nodes returns 1 node currently.  It returns 3 when we have the scaling turned up to 3 on the azure portal.  get ns returns 9 namespaces, our 4 environments (dev, test, uat, prod), azure-system, default, kube-public, kube-system, and tiller  
  > **username_0**: @username_2 additionally, we hired a contractor that set up this environment, so I'm not sure everything he did.  He provided us documentation that includes kubectl commands that were run to set things up, but I think he was just following advice from microsoft and things he googled, as he didn't seem to have any expertise in kubernetes.  
  > **username_0**: @username_1 the firewall settings seemed to be at a higher level, but I've been messing with them to try to get things working.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2018-11-12. Score: 1 | Accepted answer
  
  With help from a Microsoft Engineer, we were able to resolve our issue.
  
  The trick to it was that the node that could communicate with the outside world was always the one that the kubernetes tunnel pod was running on. This indicated a problem with communication between nodes. The traceroute indicated that this, with no network security group traffic restrictions, was bouncing between the node1 ip, to the load balancer ip, back to the node2 ip.
  
  we had a security rule that restricted all traffic that was not on our specifically approved list.
  
  One of the debug steps I had taken was to open all traffic between the load balancer ip to the load balancer ip. However, this failed to consider the fact that the traffic was not from the load balancer to the load balancer, but from nodex to the load balancer then from the load balancer to nodey.
  
  We added an inbound and outbound traffic rule that allowed traffic from the subnet ips that nodes can possibly be on to the load balancer, as well as within the same subnet for good meansure. This resolved the issue.
  
  If that's confusing, I can provide images that show the new rules and explain it a little clearer, but I'm hoping that is clear enough for anyone who experiences a similar issue.
  
  
  ---
  
  ### Answer by username_2 on 2018-10-29. Score: 0
  
  What about creating a new managed k8s cluster and start from scratch? This is what I do if the architecture is a maze. Managed k8s on azure and gcp is a couple of clicks away.
  
  > **Comments:**
  > **username_0**: We have a support ticket oustanding to microsoft, but if they don't get back to us soon, I'm going to try this.  
  > **username_2**: You could already start. You could leave the current cluster running and create a new managed cluster next to it without destroying the existing one.  
  > **username_0**: more of a matter of time than anything else.  I have a lot of other responsibilities to take care of, and technically this is microsoft's problem at the moment.  
  
  ---
  
  ### Answer by username_3 on 2024-07-02. Score: 0
  
  We had the same issue; pods on one node couldn't communicate with pods on another.
  
  ## Testing Connectivity
  
  We first confirmed our theory that pods couldn't communicate by using `kubectl exec -it <podname> -- /bin/bash` to connect to a pod on one node ([related docs](https://kubernetes.io/docs/tasks/debug/debug-application/get-shell-running-container/#getting-a-shell-to-a-container)).
  From that pod we then tried to connect to open ports on pods on the same node and on different nodes; thus showing that we could communicate with those on the same node, but not with those on a different node / confirming our theory.
  
  Since most pods don't have utility tooling installed, we used the `(timeout 1 bash -c '</dev/tcp/10.244.1.45/8089' && echo PORT OPEN || echo PORT CLOSED) 2>/dev/null` approach to test connectivity (where `10.244.1.45` is the IP of the target pod, and 8089 the listening port). [Related notes](https://stackoverflow.com/a/19866239/361842)
  
  ## Resolving the issue
  
  In our case the issue was with the Route Table associated with our Kubernetes cluster's subnet.
  
  * Open the Azure Portal
  * Navigate to the kubernetes service
  * Navigate: `Settings` > `Networking` > `Virtual Network Integration`
  * Click on the subnet
  * Click on the route table associated with our subnet
  * Under routes you'll see each route is named after the node to which it relates. The `Next hop IP address` is the IP of that node. The `Address prefix` is the CIDR under which all pods hosted on that node should sit.
  * **This was the issue:** we found that the `address prefix` associated with one node was different to the pods addresses on that node; i.e. the address prefix was `10.244.0.0/24`, but pods under that node were in the range `10.244.1.0/24`; e.g. `10.244.1.45`.
  * We resolved the issue by adding a new route with the correct information (we gave it the same name as the invalid route, just suffixing `_fix` on the end - we could probably have just corrected the invalid entry, but adding a new route felt lower risk).
  
  Hope that helps others who may hit this issue.
  
  
  ---
  
  