<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): memory</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
BUG REPORT</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6+ae4550c", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"not a git tree", BuildDate:"2016-08-30T15:45:51Z", GoVersion:"go1.7", Compiler:"gc", Platform:"darwin/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:06:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
NAME=CoreOS<br>
ID=coreos<br>
VERSION=1122.1.0<br>
VERSION_ID=1122.1.0<br>
BUILD_ID=2016-08-11-0252<br>
PRETTY_NAME="CoreOS 1122.1.0 (MoreOS)"<br>
ANSI_COLOR="1;32"<br>
HOME_URL="<a href="https://coreos.com/" rel="nofollow">https://coreos.com/</a>"<br>
BUG_REPORT_URL="<a href="https://github.com/coreos/bugs/issues">https://github.com/coreos/bugs/issues</a>"</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):<br>
Linux xxxx 4.7.0-coreos <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Thu Aug 11 02:44:21 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
cAdvisor reporting 0 memory usage for all containers</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
cAdvisor to report actual memory usage statistics</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
curl <a href="http://localhost:4194/api/v1.1/containers" rel="nofollow">http://localhost:4194/api/v1.1/containers</a></p>
<p dir="auto">Observe that all memory blocks in the json look like this:<br>
<code class="notranslate">"memory": { "usage": 0, "cache": 0, "rss": 0, "working_set": 0, "failcnt": 0, "container_data": { "pgfault": 0, "pgmajfault": 0 }, "hierarchical_data": { "pgfault": 0, "pgmajfault": 0 } },</code></p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
This seems to be a new behavior in 1.3.6</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.4.0 with <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="180738425" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/33967" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/33967/hovercard" href="https://github.com/kubernetes/kubernetes/pull/33967">#33967</a> patch applied.</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE + custom kubernetes</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): "CoreOS 1068.8.0 (MoreOS)"</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): : 4.6.3-coreos</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
But pods seem to get lost. I.e.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
...other-pods
abcd-3456205130-83b2q           1/1       Running   0          22s"><pre class="notranslate"><code class="notranslate">kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
...other-pods
abcd-3456205130-83b2q           1/1       Running   0          22s
</code></pre></div>
<p dir="auto">So first time around it comes up ok. Now I try to delete and create this deployment.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl delete -f abcdController.json
deployment &quot;abcd&quot; deleted
kubectl create -f abcdController.json
deployment &quot;abcd&quot; created"><pre class="notranslate"><code class="notranslate">kubectl delete -f abcdController.json
deployment "abcd" deleted
kubectl create -f abcdController.json
deployment "abcd" created
</code></pre></div>
<p dir="auto">After that I can not see my pod anywhere:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
..other-pods-- but abcd is not here"><pre class="notranslate"><code class="notranslate">kubectl get pods
NAME                         READY     STATUS    RESTARTS   AGE
..other-pods-- but abcd is not here
</code></pre></div>
<p dir="auto">I can see the deployment though</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl describe deployment abcd
Name:           abcd
Namespace:      default
CreationTimestamp:  Tue, 04 Oct 2016 11:25:54 +0100
Labels:         k8s-app=abcd
            kubernetes.io/cluster-service=true
            version=v1.1.0-beta2
Selector:       k8s-app=abcd,version=v1.1.0-beta2
Replicas:       0 updated | 1 total | 0 available | 0 unavailable
StrategyType:       RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
OldReplicaSets:     &lt;none&gt;
NewReplicaSet:      &lt;none&gt;"><pre class="notranslate"><code class="notranslate">kubectl describe deployment abcd
Name:           abcd
Namespace:      default
CreationTimestamp:  Tue, 04 Oct 2016 11:25:54 +0100
Labels:         k8s-app=abcd
            kubernetes.io/cluster-service=true
            version=v1.1.0-beta2
Selector:       k8s-app=abcd,version=v1.1.0-beta2
Replicas:       0 updated | 1 total | 0 available | 0 unavailable
StrategyType:       RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
OldReplicaSets:     &lt;none&gt;
NewReplicaSet:      &lt;none&gt;
</code></pre></div>
<p dir="auto">But there is no corresponding replica set for it.</p>
<p dir="auto">Here are the logs from scheduler:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570168       1 scheduler.go:135] Failed to bind pod: default/abcd-3456205130-15x25
Oct 04 10:25:44 h-stg-core-2 docker[3806]: E1004 10:25:44.570184       1 scheduler.go:137] scheduler cache ForgetPod failed: pod state wasn't assumed but get forgotten. Pod key: default/abcd-3456205130-15x25
Oct 04 10:25:44 h-stg-core-2 docker[3806]: E1004 10:25:44.570191       1 factory.go:530] Error scheduling default abcd-3456205130-15x25: Operation cannot be fulfilled on pods/binding &quot;abcd-3456205130-15x25&quot;: pod abcd-3456205130-15x25 is being deleted, cannot be assigned to a host; retrying
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570212       1 factory.go:607] Updating pod condition for default/abcd-3456205130-15x25 to (PodScheduled==False)
Oct 04 10:25:44 h-stg-core-2 docker[3806]: W1004 10:25:44.570362       1 factory.go:544] Request for pod default/abcd-3456205130-15x25 already in flight, abandoning
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570380       1 event.go:217] Event(api.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;default&quot;, Name:&quot;abcd-3456205130-15x25&quot;, UID:&quot;e84beb50-8a1c-11e6-85c8-42010a140005&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;6090&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'FailedScheduling' Binding rejected: Operation cannot be fulfilled on pods/binding &quot;abcd-3456205130-15x25&quot;: pod abcd-3456205130-15x25 is being deleted, cannot be assigned to a host
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: E1004 10:25:46.645429       1 factory.go:176] scheduler cache UpdatePod failed: no corresponding pod abcd-3456205130-15x25 in pods of node h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: E1004 10:25:46.722604       1 factory.go:176] scheduler cache UpdatePod failed: no corresponding pod abcd-3456205130-15x25 in pods of node h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-0
Oct 04 10:25:46 h-stg-core-2 docker[3806]: (*api.Pod)(0xc421648000)(&amp;TypeMeta{Kind:,APIVersion:,})"><pre class="notranslate"><code class="notranslate"> listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570168       1 scheduler.go:135] Failed to bind pod: default/abcd-3456205130-15x25
Oct 04 10:25:44 h-stg-core-2 docker[3806]: E1004 10:25:44.570184       1 scheduler.go:137] scheduler cache ForgetPod failed: pod state wasn't assumed but get forgotten. Pod key: default/abcd-3456205130-15x25
Oct 04 10:25:44 h-stg-core-2 docker[3806]: E1004 10:25:44.570191       1 factory.go:530] Error scheduling default abcd-3456205130-15x25: Operation cannot be fulfilled on pods/binding "abcd-3456205130-15x25": pod abcd-3456205130-15x25 is being deleted, cannot be assigned to a host; retrying
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570212       1 factory.go:607] Updating pod condition for default/abcd-3456205130-15x25 to (PodScheduled==False)
Oct 04 10:25:44 h-stg-core-2 docker[3806]: W1004 10:25:44.570362       1 factory.go:544] Request for pod default/abcd-3456205130-15x25 already in flight, abandoning
Oct 04 10:25:44 h-stg-core-2 docker[3806]: I1004 10:25:44.570380       1 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"abcd-3456205130-15x25", UID:"e84beb50-8a1c-11e6-85c8-42010a140005", APIVersion:"v1", ResourceVersion:"6090", FieldPath:""}): type: 'Normal' reason: 'FailedScheduling' Binding rejected: Operation cannot be fulfilled on pods/binding "abcd-3456205130-15x25": pod abcd-3456205130-15x25 is being deleted, cannot be assigned to a host
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: E1004 10:25:46.645429       1 factory.go:176] scheduler cache UpdatePod failed: no corresponding pod abcd-3456205130-15x25 in pods of node h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: E1004 10:25:46.722604       1 factory.go:176] scheduler cache UpdatePod failed: no corresponding pod abcd-3456205130-15x25 in pods of node h-stg-wk2-1
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod name is abcd-3456205130-15x25
Oct 04 10:25:46 h-stg-core-2 docker[3806]: Pod node is h-stg-wk2-0
Oct 04 10:25:46 h-stg-core-2 docker[3806]: (*api.Pod)(0xc421648000)(&amp;TypeMeta{Kind:,APIVersion:,})
</code></pre></div>
<p dir="auto">Here are thr logs from the worker node to which the new pod was originally assigned:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.357190    3706 kubelet.go:1617] SyncLoop (ADD, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.475171    3706 kubelet.go:1627] SyncLoop (REMOVE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.475232    3706 kubelet.go:1802] Failed to delete pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;, err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.484023    3706 kubelet.go:1617] SyncLoop (ADD, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.492239    3706 kubelet.go:1627] SyncLoop (REMOVE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.492298    3706 kubelet.go:1802] Failed to delete pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;, err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.503775    3706 kubelet.go:1617] SyncLoop (ADD, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.516296    3706 kubelet.go:1627] SyncLoop (REMOVE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.516384    3706 kubelet.go:1802] Failed to delete pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;, err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.542153    3706 kubelet.go:1617] SyncLoop (ADD, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.561550    3706 kubelet.go:1627] SyncLoop (REMOVE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.561617    3706 kubelet.go:1802] Failed to delete pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;, err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.567041    3706 kubelet.go:1617] SyncLoop (ADD, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.588436    3706 kubelet.go:1627] SyncLoop (REMOVE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.588497    3706 kubelet.go:1802] Failed to delete pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;, err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.599215    3706 kubelet.go:1617] SyncLoop (ADD, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.657951    3706 docker_manager.go:1906] Need to restart pod infra container for &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot; because it is not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: E1004 10:25:44.688467    3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: E1004 10:25:44.842306    3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot; container &quot;POD&quot;: symlink  /var/log/containers/abcd-3456205130-15x25_default_POD-01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79.log: no such file or directory
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.990313    3706 docker_manager.go:2167] Determined pod ip after infra change: &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;: &quot;10.120.2.11&quot;
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.023612    3706 provider.go:119] Refreshing cache for provider: *gcp_credentials.dockerConfigKeyProvider
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.024235    3706 config.go:156] body of failing http response: &amp;{0x67b3f0 0xc421ada940 0x67b210}
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: E1004 10:25:45.024263    3706 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.024290    3706 provider.go:119] Refreshing cache for provider: *gcp_credentials.dockerConfigUrlKeyProvider
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.025291    3706 config.go:156] body of failing http response: &amp;{0x67b3f0 0xc421ebfc40 0x67b210}
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: E1004 10:25:45.025342    3706 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.688160    3706 kubelet.go:1652] SyncLoop (PLEG): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;, event: &amp;pleg.PodLifecycleEvent{ID:&quot;e84beb50-8a1c-11e6-85c8-42010a140005&quot;, Type:&quot;ContainerStarted&quot;, Data:&quot;01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79&quot;}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.575266    3706 kube_docker_client.go:309] Stop pulling image &quot;eu.gcr.io/ourproject/abcd:20160915.125444.4f55766&quot;: &quot;Status: Downloaded newer image for eu.gcr.io/ourproject/abcd:20160915.125444.4f55766&quot;
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: E1004 10:25:46.592392    3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: E1004 10:25:46.665024    3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot; container &quot;abcd&quot;: symlink  /var/log/containers/abcd-3456205130-15x25_default_abcd-5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712.log: no such file or directory
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.693803    3706 kubelet.go:1633] SyncLoop (DELETE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.701327    3706 kubelet.go:1652] SyncLoop (PLEG): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;, event: &amp;pleg.PodLifecycleEvent{ID:&quot;e84beb50-8a1c-11e6-85c8-42010a140005&quot;, Type:&quot;ContainerStarted&quot;, Data:&quot;5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712&quot;}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.765815    3706 kubelet.go:1633] SyncLoop (DELETE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.829842    3706 kubelet.go:1627] SyncLoop (REMOVE, &quot;api&quot;): &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot;
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.834246    3706 kubelet_pods.go:684] Killing unwanted pod &quot;abcd-3456205130-15x25&quot;
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.834344    3706 docker_manager.go:1497] Killing container &quot;5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712 abcd default/abcd-3456205130-15x25&quot; with 0 second grace period
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: I1004 10:25:47.125227    3706 kube_docker_client.go:309] Stop pulling image &quot;eu.gcr.io/ourproject/abcd:20160915.125444.4f55766&quot;: &quot;Status: Image is up to date for eu.gcr.io/ourproject/abcd:20160915.125444.4f55766&quot;
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: E1004 10:25:47.145497    3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: E1004 10:25:47.229814    3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod &quot;abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)&quot; container &quot;abcd&quot;: symlink  /var/log/containers/abcd-3456205130-15x25_default_abcd-890b1ce28b2e4d504143f8328671b1557afd9a2354c4935a18cc1c7fc8c7b20b.log: no such file or directory
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:48.873541    3706 docker_manager.go:1536] Container &quot;5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712 abcd default/abcd-3456205130-15x25&quot; exited after 2.039169678s
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.047405    3706 docker_manager.go:1497] Killing container &quot;01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79 default/abcd-3456205130-15x25&quot; with 0 second grace period
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.091565    3706 docker_manager.go:1536] Container &quot;01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79 default/abcd-3456205130-15x25&quot; exited after 44.115046ms
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.704178    3706 kubelet_pods.go:684] Killing unwanted pod &quot;abcd-3456205130-15x25&quot;
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.706872    3706 docker_manager.go:1497] Killing container &quot;890b1ce28b2e4d504143f8328671b1557afd9a2354c4935a18cc1c7fc8c7b20b /&quot; with 30 second grace period
"><pre class="notranslate"><code class="notranslate">Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.357190    3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.475171    3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.475232    3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.484023    3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.492239    3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.492298    3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.503775    3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.516296    3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.516384    3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.542153    3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.561550    3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.561617    3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.567041    3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.588436    3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.588497    3706 kubelet.go:1802] Failed to delete pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", err: pod not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.599215    3706 kubelet.go:1617] SyncLoop (ADD, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.657951    3706 docker_manager.go:1906] Need to restart pod infra container for "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" because it is not found
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: E1004 10:25:44.688467    3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: E1004 10:25:44.842306    3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" container "POD": symlink  /var/log/containers/abcd-3456205130-15x25_default_POD-01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79.log: no such file or directory
Oct 04 10:25:44 h-stg-wk2-1 kubelet[3706]: I1004 10:25:44.990313    3706 docker_manager.go:2167] Determined pod ip after infra change: "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)": "10.120.2.11"
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.023612    3706 provider.go:119] Refreshing cache for provider: *gcp_credentials.dockerConfigKeyProvider
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.024235    3706 config.go:156] body of failing http response: &amp;{0x67b3f0 0xc421ada940 0x67b210}
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: E1004 10:25:45.024263    3706 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.024290    3706 provider.go:119] Refreshing cache for provider: *gcp_credentials.dockerConfigUrlKeyProvider
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.025291    3706 config.go:156] body of failing http response: &amp;{0x67b3f0 0xc421ebfc40 0x67b210}
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: E1004 10:25:45.025342    3706 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
Oct 04 10:25:45 h-stg-wk2-1 kubelet[3706]: I1004 10:25:45.688160    3706 kubelet.go:1652] SyncLoop (PLEG): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", event: &amp;pleg.PodLifecycleEvent{ID:"e84beb50-8a1c-11e6-85c8-42010a140005", Type:"ContainerStarted", Data:"01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79"}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.575266    3706 kube_docker_client.go:309] Stop pulling image "eu.gcr.io/ourproject/abcd:20160915.125444.4f55766": "Status: Downloaded newer image for eu.gcr.io/ourproject/abcd:20160915.125444.4f55766"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: E1004 10:25:46.592392    3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: E1004 10:25:46.665024    3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" container "abcd": symlink  /var/log/containers/abcd-3456205130-15x25_default_abcd-5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712.log: no such file or directory
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.693803    3706 kubelet.go:1633] SyncLoop (DELETE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.701327    3706 kubelet.go:1652] SyncLoop (PLEG): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)", event: &amp;pleg.PodLifecycleEvent{ID:"e84beb50-8a1c-11e6-85c8-42010a140005", Type:"ContainerStarted", Data:"5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712"}
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.765815    3706 kubelet.go:1633] SyncLoop (DELETE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.829842    3706 kubelet.go:1627] SyncLoop (REMOVE, "api"): "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.834246    3706 kubelet_pods.go:684] Killing unwanted pod "abcd-3456205130-15x25"
Oct 04 10:25:46 h-stg-wk2-1 kubelet[3706]: I1004 10:25:46.834344    3706 docker_manager.go:1497] Killing container "5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712 abcd default/abcd-3456205130-15x25" with 0 second grace period
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: I1004 10:25:47.125227    3706 kube_docker_client.go:309] Stop pulling image "eu.gcr.io/ourproject/abcd:20160915.125444.4f55766": "Status: Image is up to date for eu.gcr.io/ourproject/abcd:20160915.125444.4f55766"
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: E1004 10:25:47.145497    3706 docker_manager.go:746] Logging security options: {key:seccomp value:unconfined msg:}
Oct 04 10:25:47 h-stg-wk2-1 kubelet[3706]: E1004 10:25:47.229814    3706 docker_manager.go:1701] Failed to create symbolic link to the log file of pod "abcd-3456205130-15x25_default(e84beb50-8a1c-11e6-85c8-42010a140005)" container "abcd": symlink  /var/log/containers/abcd-3456205130-15x25_default_abcd-890b1ce28b2e4d504143f8328671b1557afd9a2354c4935a18cc1c7fc8c7b20b.log: no such file or directory
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:48.873541    3706 docker_manager.go:1536] Container "5e1101ed8ce36529cc2013333649342d1b4e77c1010b434103cef97225bfd712 abcd default/abcd-3456205130-15x25" exited after 2.039169678s
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.047405    3706 docker_manager.go:1497] Killing container "01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79 default/abcd-3456205130-15x25" with 0 second grace period
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.091565    3706 docker_manager.go:1536] Container "01ee72d6577afa46241df7223f26cc39cd7283c73ee0da2f4b98bbef7a3d6d79 default/abcd-3456205130-15x25" exited after 44.115046ms
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.704178    3706 kubelet_pods.go:684] Killing unwanted pod "abcd-3456205130-15x25"
Oct 04 10:25:49 h-stg-wk2-1 kubelet[3706]: I1004 10:25:49.706872    3706 docker_manager.go:1497] Killing container "890b1ce28b2e4d504143f8328671b1557afd9a2354c4935a18cc1c7fc8c7b20b /" with 30 second grace period

</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
Expected my pod to appear.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Create a deployment, delete a deployment, create it again. seems to happen pretty regularly.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
Some of the original discussion is on <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="180738425" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/33967" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/33967/hovercard" href="https://github.com/kubernetes/kubernetes/pull/33967">#33967</a></p>