title: How to Configure Kubernetes in Hairpin Mode
body: <p>I'm trying to enable hairpin connections on my Kubernetes service, on GKE.</p>
  
  <p>I've tried to follow the instructions here: <a href="https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/</a> to configure my kubelet config to enable hairpin mode, but it looks like my configs are never saved, even though the edit command returns without error.</p>
  
  <p>Here is what I try to set when I edit node:</p>
  
  <pre><code>spec:
    podCIDR: 10.4.1.0/24
    providerID: gce://staging/us-east4-b/gke-cluster-staging-highmem-f36fb529-cfnv
    configSource:
      configMap:
        name: my-node-config-4kbd7d944d
        namespace: kube-system
        kubeletConfigKey: kubelet
  </code></pre>
  
  <p>Here is my node config when I describe it</p>
  
  <pre><code>Name:         my-node-config-4kbd7d944d
  Namespace:    kube-system
  Labels:       &lt;none&gt;
  Annotations:  &lt;none&gt;
  
  Data
  ====
  kubelet_config:
  ----
  {
    "kind": "KubeletConfiguration",
    "apiVersion": "kubelet.config.k8s.io/v1beta1",
    "hairpinMode": "hairpin-veth"
  }
  
  </code></pre>
  
  <p>I've tried both using "edit node" and "patch". Same result in that nothing is saved. Patch returns "no changes made."</p>
  
  <p>Here is the patch command from the tutorial:</p>
  
  <pre><code>kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"
  </code></pre>
  
  <p>I also can't find any resource on where the "hairpinMode" attribute is supposed to be set. </p>
  
  <p>Any help is appreciated!</p>
  
  <p>------------------- edit ----------------</p>
  
  <p>here is why I think hairpinning isn't working.</p>
  
  <pre><code>root@668cb9686f-dzcx8:/app# nslookup tasks-staging.[my-domain].com
  Server:     10.0.32.10
  Address:    10.0.32.10#53
  
  Non-authoritative answer:
  Name:   tasks-staging.[my-domain].com
  Address: 34.102.170.43
  
  root@668cb9686f-dzcx8:/app# curl https://[my-domain].com/python/healthz
  hello
  root@668cb9686f-dzcx8:/app# nslookup my-service.default
  Server:     10.0.32.10
  Address:    10.0.32.10#53
  
  Name:   my-service.default.svc.cluster.local
  Address: 10.0.38.76
  
  root@668cb9686f-dzcx8:/app# curl https://my-service.default.svc.cluster.local/python/healthz
  curl: (7) Failed to connect to my-service.default.svc.cluster.local port 443: Connection timed out
  </code></pre>
  
  <p>also if I issue a request to localhost from my service (not curl), it gets a "connection refused." Issuing requests to the external domain, which should get routed to the same pod, is fine though. </p>
  
  <p>I only have one service, one node, one pod, and two listening ports at the moment.</p>
  
  <p>--------------------- including deployment yaml -----------------
  Deployment</p>
  
  <pre><code>spec:
    replicas: 1
      spec:
        containers:
        - name: my-app
          ports:
          - containerPort: 8080
          - containerPort: 50001
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTPS
  </code></pre>
  
  <p>Ingress:</p>
  
  <pre><code>apiVersion: extensions/v1beta1
  kind: Ingress
  spec:
    backend:
      serviceName: my-service
      servicePort: 60000
    rules:
    - http:
        paths:
        - path: /*
          backend:
            serviceName: my-service
            servicePort: 60000
        - path: /python/*
          backend:
            serviceName: my-service
            servicePort: 60001
  </code></pre>
  
  <p>service</p>
  
  <pre><code>---
  apiVersion: v1
  kind: Service
  metadata:
    name: my-service
  spec:
    ports:
    - name: port
      port: 60000
      targetPort: 8080
    - name: python-port
      port: 60001
      targetPort: 50001
    type: NodePort
  </code></pre>
  
  <p>I'm trying to set up a multi-port application where the main program trigger a script to run through issuing a request on the local machine on a different port. (I need to run something in python but the main app is in golang.)</p>
  
  <p>It's a simple script and I'd like to avoid exposing the python endpoints with the external domain, so I don't have to worry about authentication, etc. </p>
  
  <p>-------------- requests sent from my-service in golang -------------</p>
  
  <pre><code>https://[my-domain]/health: success
  https://[my-domain]/python/healthz: success
  http://my-service.default:60000/healthz: dial tcp: lookup my-service.default on 169.254.169.254:53: no such host 
  http://my-service.default/python/healthz: dial tcp: lookup my-service.default on 169.254.169.254:53: no such host 
  http://my-service.default:60001/python/healthz: dial tcp: lookup my-service.default on 169.254.169.254:53: no such host 
  http://localhost:50001/healthz: dial tcp 127.0.0.1:50001: connect: connection refused 
  http://localhost:50001/python/healthz: dial tcp 127.0.0.1:50001: connect: connection refused 
  </code></pre>