<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):Bug</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS, this particular node is a c3.4xlarge with 60 GB gp2 root volume.</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux 4.4.26-k8s <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Fri Oct 21 05:21:13 UTC 2016 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: Kops</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
The kubelet image garbage collection is triggered because disk space falls below the minimum threshold.  Kubelet starts image deletions to recover disk space but by the end of the cleanup the node has been severely impacted with pods becoming unresponsive, restarted, and often rescheduled onto other nodes.  In this particular case you can see that CPU becomes almost zero because everything has been moved off this node.</p>
<p dir="auto">Timing of this particular failure:<br>
16:56:38.000 GC started with <code class="notranslate">image_gc_manager.go:238] [imageGCManager]: Disk usage on "/dev/xvda1" (/) is at 90% which is over the high threshold (90%). Trying to free 5766938624 bytes</code> message in logs<br>
16:56:38 See the first <code class="notranslate">image_gc_manager.go:303] [imageGCManager]: Removing image</code> message in the logs<br>
17:01:38 See the final <code class="notranslate">Removing Image</code> message in the logs which is the 8th image to be deleted<br>
17:02 By this time graphs show just about everything has been moved off this node and there is nothing left running on it.</p>
<p dir="auto">The following graphs seem to indicate that as soon as the GC has finished kubelet "wakes up" and realizes everything has been removed from this node.  In the dashboard I saw this node <code class="notranslate">Ready</code> state as <code class="notranslate">Unknown</code> instead of <code class="notranslate">True</code> during this period.  It recovers and becomes functional after this hiccup and works fine when new pods get scheduled back onto the node.</p>
<p dir="auto">As expected the IO is higher during the cleanup but other metrics look fine and don't indicate a overloaded instance.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/3944676/21147113/e2439eb6-c121-11e6-9662-c53a0c74e5e5.png"><img src="https://cloud.githubusercontent.com/assets/3944676/21147113/e2439eb6-c121-11e6-9662-c53a0c74e5e5.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/3944676/21147121/e9044002-c121-11e6-9d1a-18153c1ba9e3.png"><img src="https://cloud.githubusercontent.com/assets/3944676/21147121/e9044002-c121-11e6-9d1a-18153c1ba9e3.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/3944676/21147131/f22e8840-c121-11e6-9b57-f9d15a64fc34.png"><img src="https://cloud.githubusercontent.com/assets/3944676/21147131/f22e8840-c121-11e6-9b57-f9d15a64fc34.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
No impact to running services on node.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
This issue has happened on 3 different nodes all configured the same way.  I don't believe a GC has occurred without causing this impact.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
Discussed briefly with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/justinsb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/justinsb">@justinsb</a></p>
<p dir="auto">Just a guess but is there any chance <code class="notranslate">im.imageRecordsLock.Lock()</code> at the top of <code class="notranslate">image_gc_manager.go-&gt;freeSpace</code> causes kubelet to stop processing other requests while GC is occuring?</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<ul dir="auto">
<li>initContainers imagePullPolicy</li>
<li>initContainers required value</li>
</ul>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.3&quot;, GitCommit:&quot;35fd1363821b8432ccde55632cf9ead4c79eddbb&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-12-08T21:58:56Z&quot;, GoVersion:&quot;go1.7.4&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.3&quot;, GitCommit:&quot;35fd1363821b8432ccde55632cf9ead4c79eddbb&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-12-08T21:54:19Z&quot;, GoVersion:&quot;go1.7.4&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.3", GitCommit:"35fd1363821b8432ccde55632cf9ead4c79eddbb", GitTreeState:"clean", BuildDate:"2016-12-08T21:58:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.3", GitCommit:"35fd1363821b8432ccde55632cf9ead4c79eddbb", GitTreeState:"clean", BuildDate:"2016-12-08T21:54:19Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>: cluster/kube-up.sh</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying <code class="notranslate">imagePullPolicy</code> is a required value.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="The ReplicaSet &quot;nginx&quot; is invalid: spec.template.spec.initContainers[0].imagePullPolicy: Required value"><pre class="notranslate"><code class="notranslate">The ReplicaSet "nginx" is invalid: spec.template.spec.initContainers[0].imagePullPolicy: Required value
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="The ReplicationController &quot;nginx&quot; is invalid: spec.template.spec.initContainers[0].imagePullPolicy: Required value"><pre class="notranslate"><code class="notranslate">The ReplicationController "nginx" is invalid: spec.template.spec.initContainers[0].imagePullPolicy: Required value
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">The create should succeed and the Init Container should receive a default <code class="notranslate">imagePullPolicy</code>.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Try to create either of the following:</p>
<p dir="auto"><strong>ReplicaSet</strong></p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
  name: nginx
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
            {
                &quot;name&quot;: &quot;install&quot;,
                &quot;image&quot;: &quot;busybox&quot;,
                &quot;command&quot;: [&quot;wget&quot;, &quot;-O&quot;, &quot;/work-dir/index.html&quot;, &quot;http://kubernetes.io/index.html&quot;],
                &quot;volumeMounts&quot;: [
                    {
                        &quot;name&quot;: &quot;workdir&quot;,
                        &quot;mountPath&quot;: &quot;/work-dir&quot;
                    }
                ]
            }
        ]'
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: workdir
          mountPath: /usr/share/nginx/html
      dnsPolicy: Default
      volumes:
      - name: workdir
        emptyDir: {}"><pre class="notranslate"><span class="pl-ent">apiVersion</span>: <span class="pl-s">extensions/v1beta1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">ReplicaSet</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">replicas</span>: <span class="pl-c1">3</span>
  <span class="pl-ent">template</span>:
    <span class="pl-ent">metadata</span>:
      <span class="pl-ent">labels</span>:
        <span class="pl-ent">app</span>: <span class="pl-s">nginx</span>
      <span class="pl-ent">annotations</span>:
        <span class="pl-ent">pod.beta.kubernetes.io/init-containers</span>: <span class="pl-s"><span class="pl-pds">'</span>[</span>
<span class="pl-s">            {</span>
<span class="pl-s">                "name": "install",</span>
<span class="pl-s">                "image": "busybox",</span>
<span class="pl-s">                "command": ["wget", "-O", "/work-dir/index.html", "http://kubernetes.io/index.html"],</span>
<span class="pl-s">                "volumeMounts": [</span>
<span class="pl-s">                    {</span>
<span class="pl-s">                        "name": "workdir",</span>
<span class="pl-s">                        "mountPath": "/work-dir"</span>
<span class="pl-s">                    }</span>
<span class="pl-s">                ]</span>
<span class="pl-s">            }</span>
<span class="pl-s">        ]<span class="pl-pds">'</span></span>
    <span class="pl-ent">spec</span>:
      <span class="pl-ent">containers</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
        <span class="pl-ent">image</span>: <span class="pl-s">nginx</span>
        <span class="pl-ent">ports</span>:
        - <span class="pl-ent">containerPort</span>: <span class="pl-c1">80</span>
        <span class="pl-ent">volumeMounts</span>:
        - <span class="pl-ent">name</span>: <span class="pl-s">workdir</span>
          <span class="pl-ent">mountPath</span>: <span class="pl-s">/usr/share/nginx/html</span>
      <span class="pl-ent">dnsPolicy</span>: <span class="pl-s">Default</span>
      <span class="pl-ent">volumes</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">workdir</span>
        <span class="pl-ent">emptyDir</span>: <span class="pl-s">{}</span></pre></div>
<p dir="auto"><strong>Replication Controller</strong></p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
            {
                &quot;name&quot;: &quot;install&quot;,
                &quot;image&quot;: &quot;busybox&quot;,
                &quot;command&quot;: [&quot;wget&quot;, &quot;-O&quot;, &quot;/work-dir/index.html&quot;, &quot;http://kubernetes.io/index.html&quot;],
                &quot;volumeMounts&quot;: [
                    {
                        &quot;name&quot;: &quot;workdir&quot;,
                        &quot;mountPath&quot;: &quot;/work-dir&quot;
                    }
                ]
            }
        ]'
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - name: workdir
          mountPath: /usr/share/nginx/html
      dnsPolicy: Default
      volumes:
      - name: workdir
        emptyDir: {}"><pre class="notranslate"><span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">ReplicationController</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">replicas</span>: <span class="pl-c1">3</span>
  <span class="pl-ent">selector</span>:
    <span class="pl-ent">app</span>: <span class="pl-s">nginx</span>
  <span class="pl-ent">template</span>:
    <span class="pl-ent">metadata</span>:
      <span class="pl-ent">labels</span>:
        <span class="pl-ent">app</span>: <span class="pl-s">nginx</span>
      <span class="pl-ent">annotations</span>:
        <span class="pl-ent">pod.beta.kubernetes.io/init-containers</span>: <span class="pl-s"><span class="pl-pds">'</span>[</span>
<span class="pl-s">            {</span>
<span class="pl-s">                "name": "install",</span>
<span class="pl-s">                "image": "busybox",</span>
<span class="pl-s">                "command": ["wget", "-O", "/work-dir/index.html", "http://kubernetes.io/index.html"],</span>
<span class="pl-s">                "volumeMounts": [</span>
<span class="pl-s">                    {</span>
<span class="pl-s">                        "name": "workdir",</span>
<span class="pl-s">                        "mountPath": "/work-dir"</span>
<span class="pl-s">                    }</span>
<span class="pl-s">                ]</span>
<span class="pl-s">            }</span>
<span class="pl-s">        ]<span class="pl-pds">'</span></span>
    <span class="pl-ent">spec</span>:
      <span class="pl-ent">containers</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
        <span class="pl-ent">image</span>: <span class="pl-s">nginx</span>
        <span class="pl-ent">ports</span>:
        - <span class="pl-ent">containerPort</span>: <span class="pl-c1">80</span>
        <span class="pl-ent">volumeMounts</span>:
        - <span class="pl-ent">name</span>: <span class="pl-s">workdir</span>
          <span class="pl-ent">mountPath</span>: <span class="pl-s">/usr/share/nginx/html</span>
      <span class="pl-ent">dnsPolicy</span>: <span class="pl-s">Default</span>
      <span class="pl-ent">volumes</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">workdir</span>
        <span class="pl-ent">emptyDir</span>: <span class="pl-s">{}</span></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">If I add <code class="notranslate">"imagePullPolicy": "IfNotPresent",</code> to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds.</p>
<p dir="auto">When I create an Init Container without <code class="notranslate">imagePullPolicy</code> in a StatefulSet Pod template, it succeeds.</p>
<p dir="auto">When I create an Init Container without <code class="notranslate">imagePullPolicy</code> in a bare Pod, it succeeds:</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="apiVersion: v1
kind: Pod
metadata:
  name: nginx
  annotations:
    pod.beta.kubernetes.io/init-containers: '[
        {
            &quot;name&quot;: &quot;install&quot;,
            &quot;image&quot;: &quot;busybox&quot;,
            &quot;command&quot;: [&quot;wget&quot;, &quot;-O&quot;, &quot;/work-dir/index.html&quot;, &quot;http://kubernetes.io/index.html&quot;],
            &quot;volumeMounts&quot;: [
                {
                    &quot;name&quot;: &quot;workdir&quot;,
                    &quot;mountPath&quot;: &quot;/work-dir&quot;
                }
            ]
        }
    ]'
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
    volumeMounts:
    - name: workdir
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default
  volumes:
  - name: workdir
    emptyDir: {}"><pre class="notranslate"><span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">Pod</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
  <span class="pl-ent">annotations</span>:
    <span class="pl-ent">pod.beta.kubernetes.io/init-containers</span>: <span class="pl-s"><span class="pl-pds">'</span>[</span>
<span class="pl-s">        {</span>
<span class="pl-s">            "name": "install",</span>
<span class="pl-s">            "image": "busybox",</span>
<span class="pl-s">            "command": ["wget", "-O", "/work-dir/index.html", "http://kubernetes.io/index.html"],</span>
<span class="pl-s">            "volumeMounts": [</span>
<span class="pl-s">                {</span>
<span class="pl-s">                    "name": "workdir",</span>
<span class="pl-s">                    "mountPath": "/work-dir"</span>
<span class="pl-s">                }</span>
<span class="pl-s">            ]</span>
<span class="pl-s">        }</span>
<span class="pl-s">    ]<span class="pl-pds">'</span></span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">containers</span>:
  - <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
    <span class="pl-ent">image</span>: <span class="pl-s">nginx</span>
    <span class="pl-ent">ports</span>:
    - <span class="pl-ent">containerPort</span>: <span class="pl-c1">80</span>
    <span class="pl-ent">volumeMounts</span>:
    - <span class="pl-ent">name</span>: <span class="pl-s">workdir</span>
      <span class="pl-ent">mountPath</span>: <span class="pl-s">/usr/share/nginx/html</span>
  <span class="pl-ent">dnsPolicy</span>: <span class="pl-s">Default</span>
  <span class="pl-ent">volumes</span>:
  - <span class="pl-ent">name</span>: <span class="pl-s">workdir</span>
    <span class="pl-ent">emptyDir</span>: <span class="pl-s">{}</span></pre></div>