Answers: <p>Sounds like the problem has been resolved by deleting all of the <code>kube-proxy</code> pods, which is an aggressive solution but if it works then good work!</p>
  
  <p>Best I can offer you is a collection of links about diagnosing Kubernetes problems:</p>
  
  <ul>
  <li><a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/" rel="nofollow noreferrer">Troubleshoot Applications</a></li>
  <li><a href="http://Troubleshooting%20Kubernetes%20Networking%20Issues" rel="nofollow noreferrer">Troubleshooting Kubernetes Networking Issues</a></li>
  <li><a href="https://cloud.google.com/kubernetes-engine/docs/troubleshooting" rel="nofollow noreferrer">Troubleshooting GKE</a> (GKE Specific but still useful)</li>
  <li><a href="https://www.pulumi.com/blog/kubespy-trace-a-real-time-view-into-the-heart-of-a-kubernetes-service/" rel="nofollow noreferrer">Deploy Kubespy</a></li>
  </ul>
  
  <p>Not a great answer I'm afraid, but we may have missed the opportunity to diagnose the issue.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-10-23T15:00:48.373Z
Answers: 2019-10-23T15:00:48.373Z
Answers: Richard Slater
Body: <p>I have an in house 5 node cluster running on bare-metal and I am using Calico. The cluster was working for 22 days but suddenly it stopped working. After investigating the problem I found out that the service to pod communication is broken while all the components are up and kubectl is working without a problem.</p>
  
  <p>From within the cluster (component A) if I try to curl another component (<code>bridge</code>) with its IP it works:</p>
  
  <pre><code>$ curl -vvv http://10.4.130.184:9998
  * Rebuilt URL to: http://10.4.130.184:9998/
  *   Trying 10.4.130.184...
  * TCP_NODELAY set
  * Connected to 10.4.130.184 (10.4.130.184) port 9998 (#0)
  &gt; GET / HTTP/1.1
  &gt; Host: 10.4.130.184:9998
  &gt; User-Agent: curl/7.58.0
  &gt; Accept: */*
  &gt; 
  &lt; HTTP/1.1 200 OK
  &lt; X-Powered-By: Express
  &lt; Accept-Ranges: bytes
  &lt; Cache-Control: public, max-age=0
  &lt; Last-Modified: Mon, 08 Apr 2019 14:06:42 GMT
  &lt; ETag: W/"179-169fd45c550"
  &lt; Content-Type: text/html; charset=UTF-8
  &lt; Content-Length: 377
  &lt; Date: Wed, 23 Oct 2019 09:56:35 GMT
  &lt; Connection: keep-alive
  &lt; 
  &lt;!doctype html&gt;
  &lt;html lang="en"&gt;
  &lt;head&gt;
      &lt;meta charset="utf-8" /&gt;
      &lt;meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" /&gt;
  
      &lt;title&gt;Bridge&lt;/title&gt;
  
      &lt;meta content='width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0' name='viewport' /&gt;
      &lt;meta name="viewport" content="width=device-width" /&gt;
  &lt;/head&gt;
  &lt;body&gt;
      &lt;h1&gt;Bridge&lt;/h1&gt;
  &lt;/body&gt;
  
  &lt;/html&gt;
  * Connection #0 to host 10.4.130.184 left intact
  </code></pre>
  
  <p>ns lookup for the service is also working (it resolve to the service IP):</p>
  
  <pre><code>$ nslookup bridge
  Server:    10.5.0.10
  Address 1: 10.5.0.10 kube-dns.kube-system.svc.k8s.local
  
  Name:      bridge
  Address 1: 10.5.160.50 bridge.170.svc.k8s.local
  </code></pre>
  
  <p>But the service to pod communication is broken and when I curl to the service name most of the times (60-70%) it stuck:</p>
  
  <pre><code>$ curl -vvv http://bridge:9998
  * Rebuilt URL to: http://bridge:9998/
  * Could not resolve host: bridge
  * Closing connection 0
  curl: (6) Could not resolve host: bridge
  </code></pre>
  
  <p>When I check the endpoints of the that service I can see that the IP of that pod is there:</p>
  
  <pre><code>$ kubectl get ep -n 170 bridge
  NAME     ENDPOINTS                                               AGE
  bridge   10.4.130.184:9226,10.4.130.184:9998,10.4.130.184:9226   11d
  </code></pre>
  
  <p>But as I said the curl (and any other method) that uses the service name is not working. And this is the service description:</p>
  
  <pre><code>$ kubectl describe svc -n 170 bridge
  Name:              bridge
  Namespace:         170
  Labels:            io.kompose.service=bridge
  Annotations:       Process: bridge
  Selector:          io.kompose.service=bridge
  Type:              ClusterIP
  IP:                10.5.160.50
  Port:              9998  9998/TCP
  TargetPort:        9998/TCP
  Endpoints:         10.4.130.184:9998
  Port:              9226  9226/TCP
  TargetPort:        9226/TCP
  Endpoints:         10.4.130.184:9226
  Port:              9226-udp  9226/UDP
  TargetPort:        9226/UDP
  Endpoints:         10.4.130.184:9226
  Session Affinity:  None
  Events:            &lt;none&gt;
  </code></pre>
  
  <p>This problem is not limited to just this component and it is like this for all the components.</p>
  
  <p>I restarted the CoreDNS (deleted its pods) but it is still the same. I was facing this problem before and previous time I thought it is related to Weavenet that I was using and I needed the cluster so I tore down the cluster and rebuilt it with Calico but now I am sure that this is not related to CNI and it is something else. </p>
  
  <p><strong>Environment</strong>:
  - Kubernetes version (use <code>kubectl version</code>):</p>
  
  <pre><code>Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  </code></pre>
  
  <ul>
  <li><p>Cloud provider or hardware configuration:
  This is a bare-metal cluster of 5 nodes, 1 master and 4 workers. All nodes are running Ubuntu 18.04 and they are connecting to the same subnet.</p></li>
  <li><p>OS (e.g: <code>cat /etc/os-release</code>):</p></li>
  </ul>
  
  <pre><code>NAME="Ubuntu"
  VERSION="18.04.2 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.2 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic
  </code></pre>
  
  <ul>
  <li>Kernel (e.g. <code>uname -a</code>):</li>
  </ul>
  
  <pre><code>Linux serflex-argus-1 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  </code></pre>
  
  <ul>
  <li><p>Install tools:
  Kubeadm</p></li>
  <li><p>Network plugin and version (if this is a network-related bug):
  Calico
  "cniVersion": "0.3.1"</p></li>
  </ul>
  
  <h1>Update</h1>
  
  <p>After deleting all the kube-proxy pods problem seemed to be solved but I still like to know what caused this problem. Btw I didn't see any error in the kube-proxy logs.</p>
  
  <h1>Update 2</h1>
  
  <p>The journalctl also doesn't show any strange activity, it just shows some logs when I deleted the pod of kube-proxy:</p>
  
  <pre><code>giant-7:~$ sudo journalctl --since "4 days ago" | grep kube-proxy
  
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.751108   31145 reconciler.go:181] operationExecutor.UnmountVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea")
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.751237   31145 reconciler.go:181] operationExecutor.UnmountVolume started for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea")
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: W1023 12:24:16.765057   31145 empty_dir.go:421] Warning: Failed to clear quota on /var/lib/kubelet/pods/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea/volumes/kubernetes.io~configmap/kube-proxy: ClearQuota called, but quotas disabled
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.782557   31145 operation_generator.go:831] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy" (OuterVolumeSpecName: "kube-proxy") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea"). InnerVolumeSpecName "kube-proxy". PluginName "kubernetes.io/configmap", VolumeGidValue ""
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.840793   31145 operation_generator.go:831] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq" (OuterVolumeSpecName: "kube-proxy-token-4t5tq") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea"). InnerVolumeSpecName "kube-proxy-token-4t5tq". PluginName "kubernetes.io/secret", VolumeGidValue ""
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.851656   31145 reconciler.go:301] Volume detached for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq") on node "serflex-giant-7" DevicePath ""
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.851679   31145 reconciler.go:301] Volume detached for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy") on node "serflex-giant-7" DevicePath ""
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973757   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-kube-proxy") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973826   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-lib-modules") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973958   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-kube-proxy-token-4t5tq") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.974027   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-xtables-lock") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:26 serflex-giant-7 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b/volumes/kubernetes.io~secret/kube-proxy-token-4t5tq.
  Oct 23 12:24:26 serflex-giant-7 kubelet[31145]: E1023 12:24:26.645571   31145 kuberuntime_manager.go:920] PodSandboxStatus of sandbox "ff4c1ba15b8c11a4fe86974286d81ebe4870d9e670226234fae6c7c21ce36c1d" for pod "kube-proxy-qpj4h_kube-system(4e7f5d97-fd49-461b-ae38-6bc5e3f0462b)" error: rpc error: code = Unknown desc = Error: No such container: ff4c1ba15b8c11a4fe86974286d81ebe4870d9e670226234fae6c7c21ce36c1d
  </code></pre>
  
Comments: CC BY-SA 4.0
Comments: 2019-10-23T13:05:08.990Z
Comments: Is there anything in `/var/log/kube-proxy.log` logs that would suggest `kube-proxy` had somehow failed. In addition, you can check iptables rules are being written by `kube-proxy` using the command: `iptables-save | grep hostnames`
Comments: Richard Slater
Comments: CC BY-SA 4.0
Comments: 2019-10-23T13:36:27.370Z
Comments: That log file doesn't exist: `cat: /var/log/kube-proxy.log: No such file or directory` and also the `sudo iptables-save | grep hostnames` returns nothing.
Comments: AVarf
Comments: CC BY-SA 4.0
Comments: 2019-10-23T14:32:18.460Z
Comments: Hmm... googling about a bit suggest that means you are on an OS that is using `systemd` in which case you need to use `journalctl` to view the logs (I think).  If there are no iptables then I think that means kube-proxy is working in userspace, I'm going to struggle to help with this one as my Kubernetes experience is limited to non-SystemD and kernel mode kube-proxy.  Does `iptables-save` return anything?
Comments: Richard Slater
Comments: CC BY-SA 4.0
Comments: 2019-10-23T14:49:55.750Z
Comments: I updated my question with the result of `journalctl` but I don't see anything strange. `iptables-save` returns pages of rules but when I grep for `hostnames` nothing is there.
Comments: AVarf
Comments: CC BY-SA 4.0
Comments: 2019-10-23T14:55:10.600Z
Comments: I tend to run through [Debugging Services](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/) when I run into problems that can't trivially be explained from the logs; there is a section in there on using `iptables-save` to identify if rules are not being created by `kube-proxy`.
Comments: Richard Slater
Comments: CC BY-SA 4.0
Comments: 2019-10-23T14:56:36.863Z
Comments: Thank you I will take a look.
Comments: AVarf
Tags: <kubernetes><dns>
Title: service-to-pod communication is broken in kubernetes
UserDisplayName: AVarf
ThreadText: # service-to-pod communication is broken in kubernetes
  
  (Asked by: username_0 on 2019-10-23)
  
  I have an in house 5 node cluster running on bare-metal and I am using Calico. The cluster was working for 22 days but suddenly it stopped working. After investigating the problem I found out that the service to pod communication is broken while all the components are up and kubectl is working without a problem.
  
  From within the cluster (component A) if I try to curl another component (`bridge`) with its IP it works:
  
  ```
  $ curl -vvv http://10.4.130.184:9998
  * Rebuilt URL to: http://10.4.130.184:9998/
  *   Trying 10.4.130.184...
  * TCP_NODELAY set
  * Connected to 10.4.130.184 (10.4.130.184) port 9998 (#0)
  > GET / HTTP/1.1
  > Host: 10.4.130.184:9998
  > User-Agent: curl/7.58.0
  > Accept: */*
  > 
  < HTTP/1.1 200 OK
  < X-Powered-By: Express
  < Accept-Ranges: bytes
  < Cache-Control: public, max-age=0
  < Last-Modified: Mon, 08 Apr 2019 14:06:42 GMT
  < ETag: W/"179-169fd45c550"
  < Content-Type: text/html; charset=UTF-8
  < Content-Length: 377
  < Date: Wed, 23 Oct 2019 09:56:35 GMT
  < Connection: keep-alive
  < 
  <!doctype html>
  <html lang="en">
  <head>
      <meta charset="utf-8" />
      <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
  
      <title>Bridge</title>
  
      <meta content='width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0' name='viewport' />
      <meta name="viewport" content="width=device-width" />
  </head>
  <body>
      <h1>Bridge</h1>
  </body>
  
  </html>
  * Connection #0 to host 10.4.130.184 left intact
  
  ```
  
  ns lookup for the service is also working (it resolve to the service IP):
  
  ```
  $ nslookup bridge
  Server:    10.5.0.10
  Address 1: 10.5.0.10 kube-dns.kube-system.svc.k8s.local
  
  Name:      bridge
  Address 1: 10.5.160.50 bridge.170.svc.k8s.local
  
  ```
  
  But the service to pod communication is broken and when I curl to the service name most of the times (60-70%) it stuck:
  
  ```
  $ curl -vvv http://bridge:9998
  * Rebuilt URL to: http://bridge:9998/
  * Could not resolve host: bridge
  * Closing connection 0
  curl: (6) Could not resolve host: bridge
  
  ```
  
  When I check the endpoints of the that service I can see that the IP of that pod is there:
  
  ```
  $ kubectl get ep -n 170 bridge
  NAME     ENDPOINTS                                               AGE
  bridge   10.4.130.184:9226,10.4.130.184:9998,10.4.130.184:9226   11d
  
  ```
  
  But as I said the curl (and any other method) that uses the service name is not working. And this is the service description:
  
  ```
  $ kubectl describe svc -n 170 bridge
  Name:              bridge
  Namespace:         170
  Labels:            io.kompose.service=bridge
  Annotations:       Process: bridge
  Selector:          io.kompose.service=bridge
  Type:              ClusterIP
  IP:                10.5.160.50
  Port:              9998  9998/TCP
  TargetPort:        9998/TCP
  Endpoints:         10.4.130.184:9998
  Port:              9226  9226/TCP
  TargetPort:        9226/TCP
  Endpoints:         10.4.130.184:9226
  Port:              9226-udp  9226/UDP
  TargetPort:        9226/UDP
  Endpoints:         10.4.130.184:9226
  Session Affinity:  None
  Events:            <none>
  
  ```
  
  This problem is not limited to just this component and it is like this for all the components.
  
  I restarted the CoreDNS (deleted its pods) but it is still the same. I was facing this problem before and previous time I thought it is related to Weavenet that I was using and I needed the cluster so I tore down the cluster and rebuilt it with Calico but now I am sure that this is not related to CNI and it is something else.
  
  **Environment**:
  - Kubernetes version (use `kubectl version`):
  
  ```
  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
  
  ```
  
  * Cloud provider or hardware configuration:
    This is a bare-metal cluster of 5 nodes, 1 master and 4 workers. All nodes are running Ubuntu 18.04 and they are connecting to the same subnet.
  * OS (e.g: `cat /etc/os-release`):
  
  ```
  NAME="Ubuntu"
  VERSION="18.04.2 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.2 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic
  
  ```
  
  * Kernel (e.g. `uname -a`):
  
  ```
  Linux serflex-argus-1 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  
  ```
  
  * Install tools:
    Kubeadm
  * Network plugin and version (if this is a network-related bug):
    Calico
    "cniVersion": "0.3.1"
  
  # Update
  
  After deleting all the kube-proxy pods problem seemed to be solved but I still like to know what caused this problem. Btw I didn't see any error in the kube-proxy logs.
  
  # Update 2
  
  The journalctl also doesn't show any strange activity, it just shows some logs when I deleted the pod of kube-proxy:
  
  ```
  giant-7:~$ sudo journalctl --since "4 days ago" | grep kube-proxy
  
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.751108   31145 reconciler.go:181] operationExecutor.UnmountVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea")
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.751237   31145 reconciler.go:181] operationExecutor.UnmountVolume started for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea")
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: W1023 12:24:16.765057   31145 empty_dir.go:421] Warning: Failed to clear quota on /var/lib/kubelet/pods/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea/volumes/kubernetes.io~configmap/kube-proxy: ClearQuota called, but quotas disabled
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.782557   31145 operation_generator.go:831] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy" (OuterVolumeSpecName: "kube-proxy") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea"). InnerVolumeSpecName "kube-proxy". PluginName "kubernetes.io/configmap", VolumeGidValue ""
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.840793   31145 operation_generator.go:831] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq" (OuterVolumeSpecName: "kube-proxy-token-4t5tq") pod "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea" (UID: "02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea"). InnerVolumeSpecName "kube-proxy-token-4t5tq". PluginName "kubernetes.io/secret", VolumeGidValue ""
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.851656   31145 reconciler.go:301] Volume detached for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy-token-4t5tq") on node "serflex-giant-7" DevicePath ""
  Oct 23 12:24:16 serflex-giant-7 kubelet[31145]: I1023 12:24:16.851679   31145 reconciler.go:301] Volume detached for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/02a1a1fb-2411-4f0e-98e1-ef2dbd6149ea-kube-proxy") on node "serflex-giant-7" DevicePath ""
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973757   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-kube-proxy") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973826   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-lib-modules") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.973958   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-4t5tq" (UniqueName: "kubernetes.io/secret/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-kube-proxy-token-4t5tq") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:25 serflex-giant-7 kubelet[31145]: I1023 12:24:25.974027   31145 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b-xtables-lock") pod "kube-proxy-qpj4h" (UID: "4e7f5d97-fd49-461b-ae38-6bc5e3f0462b")
  Oct 23 12:24:26 serflex-giant-7 systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/4e7f5d97-fd49-461b-ae38-6bc5e3f0462b/volumes/kubernetes.io~secret/kube-proxy-token-4t5tq.
  Oct 23 12:24:26 serflex-giant-7 kubelet[31145]: E1023 12:24:26.645571   31145 kuberuntime_manager.go:920] PodSandboxStatus of sandbox "ff4c1ba15b8c11a4fe86974286d81ebe4870d9e670226234fae6c7c21ce36c1d" for pod "kube-proxy-qpj4h_kube-system(4e7f5d97-fd49-461b-ae38-6bc5e3f0462b)" error: rpc error: code = Unknown desc = Error: No such container: ff4c1ba15b8c11a4fe86974286d81ebe4870d9e670226234fae6c7c21ce36c1d
  
  ```
  
  **Tags:** kubernetes, dns
  
  > **Comments:**
  > **username_1**: Is there anything in `/var/log/kube-proxy.log` logs that would suggest `kube-proxy` had somehow failed. In addition, you can check iptables rules are being written by `kube-proxy` using the command: `iptables-save | grep hostnames`  
  > **username_0**: That log file doesn't exist: `cat: /var/log/kube-proxy.log: No such file or directory` and also the `sudo iptables-save | grep hostnames` returns nothing.  
  > **username_1**: Hmm... googling about a bit suggest that means you are on an OS that is using `systemd` in which case you need to use `journalctl` to view the logs (I think).  If there are no iptables then I think that means kube-proxy is working in userspace, I'm going to struggle to help with this one as my Kubernetes experience is limited to non-SystemD and kernel mode kube-proxy.  Does `iptables-save` return anything?  
  > **username_0**: I updated my question with the result of `journalctl` but I don't see anything strange. `iptables-save` returns pages of rules but when I grep for `hostnames` nothing is there.  
  > **username_1**: I tend to run through [Debugging Services](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/) when I run into problems that can't trivially be explained from the logs; there is a section in there on using `iptables-save` to identify if rules are not being created by `kube-proxy`.  
  > **username_0**: Thank you I will take a look.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-10-23. Score: 1
  
  Sounds like the problem has been resolved by deleting all of the `kube-proxy` pods, which is an aggressive solution but if it works then good work!
  
  Best I can offer you is a collection of links about diagnosing Kubernetes problems:
  
  * [Troubleshoot Applications](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/)
  * [Troubleshooting Kubernetes Networking Issues](http://Troubleshooting%20Kubernetes%20Networking%20Issues)
  * [Troubleshooting GKE](https://cloud.google.com/kubernetes-engine/docs/troubleshooting) (GKE Specific but still useful)
  * [Deploy Kubespy](https://www.pulumi.com/blog/kubespy-trace-a-real-time-view-into-the-heart-of-a-kubernetes-service/)
  
  Not a great answer I'm afraid, but we may have missed the opportunity to diagnose the issue.
  
  
  ---
  
  