<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): "read-only"</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): Bug report. I am not sure if this is an issue with Kubernetes, Docker, or Google Cloud, but posting here to get this started.</p>
<p dir="auto">I've looked at 29358, 29166, 28750, 29903 but they all appear to be related to different issues.</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.6&quot;, GitCommit:&quot;ae4550cc9c89a593bcda6678df201db1b208133b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-26T18:13:23Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.6&quot;, GitCommit:&quot;ae4550cc9c89a593bcda6678df201db1b208133b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-26T18:06:06Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:06:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Google Cloud (GKE)</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
In container:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME=&quot;Alpine Linux&quot;
ID=alpine
VERSION_ID=3.4.0
PRETTY_NAME=&quot;Alpine Linux v3.4&quot;
HOME_URL=&quot;http://alpinelinux.org&quot;
BUG_REPORT_URL=&quot;http://bugs.alpinelinux.org&quot;"><pre class="notranslate"><code class="notranslate">NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.4.0
PRETTY_NAME="Alpine Linux v3.4"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"
</code></pre></div>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):<br>
In container:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Linux redis-cache 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 Linux"><pre class="notranslate"><code class="notranslate">Linux redis-cache 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 Linux
</code></pre></div>
<ul dir="auto">
<li><strong>Install tools</strong>: Google Cloud SDK</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
When kubernetes moves around pods, sometimes the pods with <code class="notranslate">gcePersistentDisk</code> volumes end up not coming up ("timeout expired waiting for volumes to attach/mount for pod"). When this doesn't resolve it self, I delete the pod. Kubernetes recreates, eventually mounting the disk, but intermittently the disk ends up being mounted read-only even though it is the only container connected to it and <code class="notranslate">kubectl describe pod</code> shows the volume as "ReadWrite".</p>
<p dir="auto">To resolve this, I have to delete the <em>deployment</em> and wait for the disk to auto-detach, or use <code class="notranslate">glcoud compute instance detach-disk</code> to remove it from the instance. After doing this, re-creating the deployment seems to launch the pod correctly and the drive gets attached in read-write mode as it should.</p>
<p dir="auto">I've talked with another team that has seen the same behavior (<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/davidewatson/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/davidewatson">@davidewatson</a>).</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
A persisted disk assigned to a single node should be read-write within the container.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
It's intermittent. :/</p>
<p dir="auto">In our case we see this when attaching disks to redis instances. The disks are not huge (200G of which maybe 15G is used) but the redis instance (PID 1 in the container) immediately launches and tries to read the disk so it would need to be mounted on container start.</p>
<p dir="auto">We have a fairly high rate of mount timeouts that never resolve. Perhaps 20% of the time. I've let some of them run as long as an hour or a day and they just keep trying to remount. Sometimes deleting the pod resolves this. Sometimes deleting the pod results in this read-only mount.</p>
<p dir="auto">Deployment spec:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis-cache
spec:
  template:
    metadata:
      labels:
        purpose: cache
        service: redis
    spec:
      volumes:
        - name: data
          gcePersistentDisk:
            pdName: redis-cache
            fsType: ext4
      containers:
      - name: redis
        image: us.gcr.io/xxxxxxxx/redis-cache:latest
        resources:
          requests:
            memory: &quot;10G&quot;
        ports:
        - containerPort: 6379
        volumeMounts:
        - mountPath: /data
          name: data
        args: [&quot;redis-server&quot;, &quot;/etc/redis.conf&quot;]"><pre class="notranslate"><code class="notranslate">apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis-cache
spec:
  template:
    metadata:
      labels:
        purpose: cache
        service: redis
    spec:
      volumes:
        - name: data
          gcePersistentDisk:
            pdName: redis-cache
            fsType: ext4
      containers:
      - name: redis
        image: us.gcr.io/xxxxxxxx/redis-cache:latest
        resources:
          requests:
            memory: "10G"
        ports:
        - containerPort: 6379
        volumeMounts:
        - mountPath: /data
          name: data
        args: ["redis-server", "/etc/redis.conf"]
</code></pre></div>
<p dir="auto"><code class="notranslate">Dockerfile</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="FROM redis:3.2-alpine
ADD ./deploy/redis-cache.conf /etc/redis.conf"><pre class="notranslate"><code class="notranslate">FROM redis:3.2-alpine
ADD ./deploy/redis-cache.conf /etc/redis.conf
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">Here's a full log of the behavior.</p>
<p dir="auto">First we saw that the redis instance was down. Inspecting the cluster showed that kubernetes had shuffled the pod.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get pods -l purpose=cache
NAME                                                            READY     STATUS              RESTARTS   AGE
redis-cache-3629295551-h5zig                            1/1       Terminating         1          2d
redis-cache-3629295551-mqkm2                            0/1       ContainerCreating   0          2m"><pre class="notranslate"><code class="notranslate">$ kubectl get pods -l purpose=cache
NAME                                                            READY     STATUS              RESTARTS   AGE
redis-cache-3629295551-h5zig                            1/1       Terminating         1          2d
redis-cache-3629295551-mqkm2                            0/1       ContainerCreating   0          2m
</code></pre></div>
<p dir="auto">About an hour later the pod still had not come up. Inspecting the pod, shows that it's been retrying the mount over the hour and not succeeding. Note that this was re-launched on the same node that the evicted pod was running on.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl describe pod redis-cache-3629295551-abfoh
Name:       redis-cache-3629295551-abfoh
Namespace:  default
Node:       gke-app-name-pool-1-b5918e16-rqip/10.138.0.6
...
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath   Type        Reason      Message
  --------- --------    -----   ----                            -------------   --------    ------      -------
  59m       59m     1   {default-scheduler }                            Normal      Scheduled   Successfully assigned redis-cache-3629295551-abfoh to gke-app-name-pool-1-b5918e16-rqip
  57m       14s     27  {kubelet gke-app-name-pool-1-b5918e16-rqip}         Warning     FailedMount Unable to mount volumes for pod &quot;redis-cache-3629295551-abfoh_default(72768d65-7c4a-11e6-a198-42010a8a000d)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;redis-cache-3629295551-abfoh&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[data]
  57m       14s     27  {kubelet gke-app-name-pool-1-b5918e16-rqip}         Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;redis-cache-3629295551-abfoh&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[data]"><pre class="notranslate"><code class="notranslate">$ kubectl describe pod redis-cache-3629295551-abfoh
Name:       redis-cache-3629295551-abfoh
Namespace:  default
Node:       gke-app-name-pool-1-b5918e16-rqip/10.138.0.6
...
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath   Type        Reason      Message
  --------- --------    -----   ----                            -------------   --------    ------      -------
  59m       59m     1   {default-scheduler }                            Normal      Scheduled   Successfully assigned redis-cache-3629295551-abfoh to gke-app-name-pool-1-b5918e16-rqip
  57m       14s     27  {kubelet gke-app-name-pool-1-b5918e16-rqip}         Warning     FailedMount Unable to mount volumes for pod "redis-cache-3629295551-abfoh_default(72768d65-7c4a-11e6-a198-42010a8a000d)": timeout expired waiting for volumes to attach/mount for pod "redis-cache-3629295551-abfoh"/"default". list of unattached/unmounted volumes=[data]
  57m       14s     27  {kubelet gke-app-name-pool-1-b5918e16-rqip}         Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "redis-cache-3629295551-abfoh"/"default". list of unattached/unmounted volumes=[data]
</code></pre></div>
<p dir="auto">I deleted the pod and let the deployment create a new one:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl delete pod redis-cache-3629295551-abfoh
pod &quot;redis-cache-3629295551-abfoh&quot; deleted
$ kubectl get pods -l purpose=cache
NAME                                                            READY     STATUS              RESTARTS   AGE
redis-cache-3629295551-abfoh                            0/1       Terminating         0          1h
redis-cache-3629295551-o7mm9                            0/1       ContainerCreating   0          7s"><pre class="notranslate"><code class="notranslate">$ kubectl delete pod redis-cache-3629295551-abfoh
pod "redis-cache-3629295551-abfoh" deleted
$ kubectl get pods -l purpose=cache
NAME                                                            READY     STATUS              RESTARTS   AGE
redis-cache-3629295551-abfoh                            0/1       Terminating         0          1h
redis-cache-3629295551-o7mm9                            0/1       ContainerCreating   0          7s
</code></pre></div>
<p dir="auto">This eventually comes up, on another node:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl describe pod redis-cache-3629295551-o7mm9
Name:       redis-cache-3629295551-o7mm9
Namespace:  default
Node:       gke-app-name-pool-1-b5918e16-8cdb/10.138.0.4
...
Volumes:
  data:
    Type:   GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName: redis-cache
    FSType: ext4
    Partition:  0
    ReadOnly:   false
...
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath       Type        ReasonMessage
  --------- --------    -----   ----                            -------------       --------    -------------
  7m        7m      1   {default-scheduler }                                Normal      Scheduled   Successfully assigned redis-cache-3629295551-o7mm9 to gke-app-name-pool-1-b5918e16-8cdb
  5m        3m      2   {kubelet gke-app-name-pool-1-b5918e16-8cdb}             Warning     FailedMount Unable to mount volumes for pod &quot;redis-cache-3629295551-o7mm9_default(d6fb24b0-7c52-11e6-a198-42010a8a000d)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;redis-cache-3629295551-o7mm9&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[data]
  5m        3m      2   {kubelet gke-app-name-pool-1-b5918e16-8cdb}             Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;redis-cache-3629295551-o7mm9&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[data]
  1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  Normal      PulledContainer image &quot;us.gcr.io/xxxxxxx/redis-cache:latest&quot; already present on machine
  1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  Normal      CreateCreated container with docker id 896b189ef167
  1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  Normal      StarteStarted container with docker id 896b189ef167"><pre class="notranslate"><code class="notranslate">$ kubectl describe pod redis-cache-3629295551-o7mm9
Name:       redis-cache-3629295551-o7mm9
Namespace:  default
Node:       gke-app-name-pool-1-b5918e16-8cdb/10.138.0.4
...
Volumes:
  data:
    Type:   GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName: redis-cache
    FSType: ext4
    Partition:  0
    ReadOnly:   false
...
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath       Type        ReasonMessage
  --------- --------    -----   ----                            -------------       --------    -------------
  7m        7m      1   {default-scheduler }                                Normal      Scheduled   Successfully assigned redis-cache-3629295551-o7mm9 to gke-app-name-pool-1-b5918e16-8cdb
  5m        3m      2   {kubelet gke-app-name-pool-1-b5918e16-8cdb}             Warning     FailedMount Unable to mount volumes for pod "redis-cache-3629295551-o7mm9_default(d6fb24b0-7c52-11e6-a198-42010a8a000d)": timeout expired waiting for volumes to attach/mount for pod "redis-cache-3629295551-o7mm9"/"default". list of unattached/unmounted volumes=[data]
  5m        3m      2   {kubelet gke-app-name-pool-1-b5918e16-8cdb}             Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "redis-cache-3629295551-o7mm9"/"default". list of unattached/unmounted volumes=[data]
  1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  Normal      PulledContainer image "us.gcr.io/xxxxxxx/redis-cache:latest" already present on machine
  1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  Normal      CreateCreated container with docker id 896b189ef167
  1m        1m      1   {kubelet gke-app-name-pool-1-b5918e16-8cdb} spec.containers{redis}  Normal      StarteStarted container with docker id 896b189ef167
</code></pre></div>
<p dir="auto">But the volume isn't writable inside the container:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl exec redis-cache-3629295551-o7mm9 -- mount | grep /data
/dev/sdb on /data type ext4 (ro,relatime,data=ordered)"><pre class="notranslate"><code class="notranslate">$ kubectl exec redis-cache-3629295551-o7mm9 -- mount | grep /data
/dev/sdb on /data type ext4 (ro,relatime,data=ordered)
</code></pre></div>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): "btrfs kubelet"</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): v1.4.6</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: bare metal</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CentOS Linux 7 (Core)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):  Linux vilnus 3.10.0-327.36.3.el7.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: Ansible script from <a href="https://github.com/kubernetes/contrib.git">https://github.com/kubernetes/contrib.git</a> with custom changes.</li>
<li><strong>Others</strong>: Cluster with 4 bare metal nodes, same OS on all nodes</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
kubelet continuously reports following error messages:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Dec 06 01:27:26 vilnus kubelet[2447]: E1206 01:27:26.798004    2447 kubelet.go:2132] Failed to check if disk space is available on the root partition: failed to get fs info for &quot;root&quot;: error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 37 in cached partitions map"><pre class="notranslate"><code class="notranslate">Dec 06 01:27:26 vilnus kubelet[2447]: E1206 01:27:26.798004    2447 kubelet.go:2132] Failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 37 in cached partitions map
</code></pre></div>
<p dir="auto">As (I assume) a side effect heapster pod fails with following error messages:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2016-12-07T22:48:05.074258000Z E1207 22:48:05.073677       1 summary.go:114] error while getting metrics summary from Kubelet antego(192.168.81.104:10255): request failed - &quot;500 Internal Server Error&quot;, response: &quot;Internal Error: failed RootFsInfo: error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 36 in cached partitions map&quot;"><pre class="notranslate"><code class="notranslate">2016-12-07T22:48:05.074258000Z E1207 22:48:05.073677       1 summary.go:114] error while getting metrics summary from Kubelet antego(192.168.81.104:10255): request failed - "500 Internal Server Error", response: "Internal Error: failed RootFsInfo: error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 36 in cached partitions map"
</code></pre></div>
<p dir="auto">When kubelet starts there is no device with major:minor ID '0:37':</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Dec 06 01:27:21 vilnus kubelet[2447]: I1206 01:27:21.304451    2447 fs.go:117] Filesystem partitions: map[/dev/sda4:{mountpoint:/ major:0 minor:35 fsType:btrfs blockSize:0} /dev/sda2:{mountp
oint:/boot major:8 minor:2 fsType:xfs blockSize:0}]"><pre class="notranslate"><code class="notranslate">Dec 06 01:27:21 vilnus kubelet[2447]: I1206 01:27:21.304451    2447 fs.go:117] Filesystem partitions: map[/dev/sda4:{mountpoint:/ major:0 minor:35 fsType:btrfs blockSize:0} /dev/sda2:{mountp
oint:/boot major:8 minor:2 fsType:xfs blockSize:0}]
</code></pre></div>
<p dir="auto">stat tool does not report device with 0:35 but 0:37 (i.e. 25h/37d):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[root@vilnus rubinstein]# stat /
  File: ‘/’
  Size: 152       	Blocks: 0          IO Block: 4096   directory
Device: 25h/37d	Inode: 256         Links: 1
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:root_t:s0
Access: 2016-12-07 23:49:42.368681876 +0100
Modify: 2016-12-06 01:27:50.993617031 +0100
Change: 2016-12-06 01:27:50.993617031 +0100
 Birth: -"><pre class="notranslate"><code class="notranslate">[root@vilnus rubinstein]# stat /
  File: ‘/’
  Size: 152       	Blocks: 0          IO Block: 4096   directory
Device: 25h/37d	Inode: 256         Links: 1
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:root_t:s0
Access: 2016-12-07 23:49:42.368681876 +0100
Modify: 2016-12-06 01:27:50.993617031 +0100
Change: 2016-12-06 01:27:50.993617031 +0100
 Birth: -
</code></pre></div>
<p dir="auto">Here is also /etc/fstab</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#
# /etc/fstab
# Created by anaconda on Tue Nov  8 15:25:05 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=4706dd7f-81aa-4bb6-bcb6-e742aae08456 /                       btrfs   subvol=root     0 0
UUID=72a8c6a4-b442-48ef-a6ad-a8da2146fab6 /boot                   xfs     defaults        0 0
UUID=039B-CD7E          /boot/efi               vfat    umask=0077,shortname=winnt 0 0
UUID=56f454ff-1d8e-4900-b034-1f777df92c30 swap                    swap    defaults        0 0
UUID=4706dd7f-81aa-4bb6-bcb6-e742aae08456 /mnt/disk-4706dd7f-81aa-4bb6-bcb6-e742aae08456 btrfs   subvol=/ 0 0"><pre class="notranslate"><code class="notranslate">#
# /etc/fstab
# Created by anaconda on Tue Nov  8 15:25:05 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=4706dd7f-81aa-4bb6-bcb6-e742aae08456 /                       btrfs   subvol=root     0 0
UUID=72a8c6a4-b442-48ef-a6ad-a8da2146fab6 /boot                   xfs     defaults        0 0
UUID=039B-CD7E          /boot/efi               vfat    umask=0077,shortname=winnt 0 0
UUID=56f454ff-1d8e-4900-b034-1f777df92c30 swap                    swap    defaults        0 0
UUID=4706dd7f-81aa-4bb6-bcb6-e742aae08456 /mnt/disk-4706dd7f-81aa-4bb6-bcb6-e742aae08456 btrfs   subvol=/ 0 0
</code></pre></div>
<p dir="auto">Same errors appears on all nodes and all nodes have / mounted to btrfs partition.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">No errors, kubernetes 1.2.0 installed from CentOS 7 package had no such errors.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Kubernetes binaries were installed from here :<br>
<a href="https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64" rel="nofollow">https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64</a></p>
<p dir="auto">Run kubelet on btrfs root partition.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">I would like at least to have recommendations how to figure out what exactly issue is.</p>