<p dir="auto">I'm trying to set up AWS ELB attach volume access and keep getting an error on the kubelet. I have some bad feeling it might be related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="88465830" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/9801" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/9801/hovercard" href="https://github.com/kubernetes/kubernetes/issues/9801">#9801</a> as these nodes are CoreOS that were brought up by Terraform with custom service files but everything else is working fine (we're running cluster monitoring, dns, and a number of our own pods without issues). I've run awscli on that instance and the privileges definitely work. What am I doing wrong?</p>
<p dir="auto">logs from kubelet service</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0719 02:22:59.549444    9662 factory.go:234] Registering Docker factory
I0719 02:22:59.549826    9662 factory.go:89] Registering Raw factory
I0719 02:22:59.638468    9662 manager.go:946] Started watching for new ooms in manager
I0719 02:22:59.639054    9662 oomparser.go:183] oomparser using systemd
I0719 02:22:59.640589    9662 manager.go:243] Starting recovery of all containers
E0719 02:22:59.679107    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
I0719 02:22:59.753816    9662 manager.go:248] Recovery completed
I0719 02:22:59.816369    9662 status_manager.go:76] Starting to sync pod status with apiserver
I0719 02:22:59.816426    9662 kubelet.go:1725] Starting kubelet main sync loop.
E0719 02:22:59.953337    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:01.047929    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:01.975409    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:03.645707    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:06.917503    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
I0719 02:23:07.790941    9662 server.go:635] POST /stats/container/: (46.254087ms) 0 [[Go 1.1 package http] 10.0.39.222:45497]"><pre class="notranslate"><code class="notranslate">I0719 02:22:59.549444    9662 factory.go:234] Registering Docker factory
I0719 02:22:59.549826    9662 factory.go:89] Registering Raw factory
I0719 02:22:59.638468    9662 manager.go:946] Started watching for new ooms in manager
I0719 02:22:59.639054    9662 oomparser.go:183] oomparser using systemd
I0719 02:22:59.640589    9662 manager.go:243] Starting recovery of all containers
E0719 02:22:59.679107    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
I0719 02:22:59.753816    9662 manager.go:248] Recovery completed
I0719 02:22:59.816369    9662 status_manager.go:76] Starting to sync pod status with apiserver
I0719 02:22:59.816426    9662 kubelet.go:1725] Starting kubelet main sync loop.
E0719 02:22:59.953337    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:01.047929    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:01.975409    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:03.645707    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
E0719 02:23:06.917503    9662 kubelet.go:787] Unable to construct api.Node object for kubelet: failed to get external ID from cloud provider: instance not found
I0719 02:23:07.790941    9662 server.go:635] POST /stats/container/: (46.254087ms) 0 [[Go 1.1 package http] 10.0.39.222:45497]
</code></pre></div>
<p dir="auto">kube-kubelet.service</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
Environment=&quot;KUBERNETES_BINARY_VERSION=1.0.0&quot;
EnvironmentFile=/etc/environment
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kubelet https://storage.googleapis.com/kubernetes-release/release/v${KUBERNETES_BINARY_VERSION}/bin/linux/amd64
ExecStartPre=/usr/bin/chmod +x /opt/bin/kubelet
ExecStart=/opt/bin/kubelet \
  --address=0.0.0.0 \
  --port=10250 \
  --cloud-provider=aws \
  --hostname-override=${COREOS_PRIVATE_IPV4} \
  --api-servers=${KUBE_MASTER_IP}:8080 \
  --allow-privileged=false \
  --cluster_dns=10.2.0.2 \
  --cluster_domain=cluster.local \
  --cadvisor_port=4194 \
  --healthz_bind_address=0.0.0.0 \
  --healthz_port=10248 \
  --v=2 \
  --logtostderr=true
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target"><pre class="notranslate"><code class="notranslate">[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
Environment="KUBERNETES_BINARY_VERSION=1.0.0"
EnvironmentFile=/etc/environment
ExecStartPre=/usr/bin/curl -L -o /opt/bin/kubelet https://storage.googleapis.com/kubernetes-release/release/v${KUBERNETES_BINARY_VERSION}/bin/linux/amd64
ExecStartPre=/usr/bin/chmod +x /opt/bin/kubelet
ExecStart=/opt/bin/kubelet \
  --address=0.0.0.0 \
  --port=10250 \
  --cloud-provider=aws \
  --hostname-override=${COREOS_PRIVATE_IPV4} \
  --api-servers=${KUBE_MASTER_IP}:8080 \
  --allow-privileged=false \
  --cluster_dns=10.2.0.2 \
  --cluster_domain=cluster.local \
  --cadvisor_port=4194 \
  --healthz_bind_address=0.0.0.0 \
  --healthz_port=10248 \
  --v=2 \
  --logtostderr=true
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
</code></pre></div>
<p dir="auto">iam policy</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;Version&quot;: &quot;2012-10-17&quot;,
  &quot;Statement&quot;: [
    {
      &quot;Effect&quot;: &quot;Allow&quot;,
      &quot;Action&quot;: [
        &quot;ec2:Describe*&quot;
      ],
      &quot;Resource&quot;: [
        &quot;*&quot;
      ]
    },
    {
      &quot;Effect&quot;: &quot;Allow&quot;,
      &quot;Action&quot;: [
        &quot;ec2:AttachVolume&quot;,
        &quot;ec2:DetachVolume&quot;
      ],
      &quot;Resource&quot;: [
        &quot;arn:aws:ec2:*:*:instance/*&quot;
      ]
    },
  ]
}"><pre class="notranslate">{
  <span class="pl-ent">"Version"</span>: <span class="pl-s"><span class="pl-pds">"</span>2012-10-17<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"Statement"</span>: [
    {
      <span class="pl-ent">"Effect"</span>: <span class="pl-s"><span class="pl-pds">"</span>Allow<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"Action"</span>: [
        <span class="pl-s"><span class="pl-pds">"</span>ec2:Describe*<span class="pl-pds">"</span></span>
      ],
      <span class="pl-ent">"Resource"</span>: [
        <span class="pl-s"><span class="pl-pds">"</span>*<span class="pl-pds">"</span></span>
      ]
    },
    {
      <span class="pl-ent">"Effect"</span>: <span class="pl-s"><span class="pl-pds">"</span>Allow<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"Action"</span>: [
        <span class="pl-s"><span class="pl-pds">"</span>ec2:AttachVolume<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>ec2:DetachVolume<span class="pl-pds">"</span></span>
      ],
      <span class="pl-ent">"Resource"</span>: [
        <span class="pl-s"><span class="pl-pds">"</span>arn:aws:ec2:*:*:instance/*<span class="pl-pds">"</span></span>
      ]
    },
  ]
}</pre></div>
<p dir="auto"><strong>BUG REPORT</strong></p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): v1.4.0</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): default GCI image</li>
<li><strong>Install tools</strong>: kube-up</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<ul dir="auto">
<li>Create TPR "MyResource"</li>
<li>Create namespace "soak-test"</li>
<li>Create a "MyResource" "a" under "soak-test":</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat my-resource-a.yaml
apiVersion: &quot;example.com/v1&quot;
kind: &quot;MyResource&quot;
metadata:
  name: &quot;a&quot;
...
$ kubectl --namespace soak-test create -f my-resource-a.yaml
$ kubectl --namespace soak-test get MyResource
NAME           LABELS    DATA
a           ..."><pre class="notranslate"><code class="notranslate">$ cat my-resource-a.yaml
apiVersion: "example.com/v1"
kind: "MyResource"
metadata:
  name: "a"
...
$ kubectl --namespace soak-test create -f my-resource-a.yaml
$ kubectl --namespace soak-test get MyResource
NAME           LABELS    DATA
a           ...
</code></pre></div>
<ul dir="auto">
<li>Delete namespace "soak-test"</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl delete ns soak-test"><pre class="notranslate"><code class="notranslate">$ kubectl delete ns soak-test
</code></pre></div>
<ul dir="auto">
<li>Verify it's deleted</li>
<li>Recreate namespace "soak-test"</li>
</ul>
<p dir="auto">Here I'm expecting that the namespace should be fresh. But previous MyResource "a" still exists.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl --namespace soak-test get MyResource
NAME           LABELS    DATA
a           ..."><pre class="notranslate"><code class="notranslate">$ kubectl --namespace soak-test get MyResource
NAME           LABELS    DATA
a           ...
</code></pre></div>
<p dir="auto">Is this expected or a bug?</p>