<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): rbd</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): Bug Report</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:06:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Baremetal (Supermicro 1028R)</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS 1068.10.0</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.6.3-coreos</li>
<li><strong>Install tools</strong>: custom</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">I first deployed a new pod (via deployment) for running Grafana, using two Ceph RBD volumes (for <code class="notranslate">/var/lib/grafana</code> and <code class="notranslate">/etc/grafana</code>).</p>
<p dir="auto">I then found that I'd mounted a volume over the default config with the <code class="notranslate">/etc/grafana</code> mount and decided to just skip that for now and remove that volume. I applied a new version of the deployment with just the RBD for <code class="notranslate">/var/lib/grafana</code>. It got created on a different node and was unable to mount the volume because it was already mounted on the first node.</p>
<p dir="auto">On investigation, I found that the original node still had the RBD volumes mapped and mounted. The kubelet logs indicate "success" when tearing down the volumes:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809860     766 reconciler.go:179] UnmountVolume operation started for volume &quot;kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06&quot; (spec.Name: &quot;default-token-a7f06&quot;) from pod &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot; (UID: &quot;6e4e5190-7617-11e6-b97e-001
5c5edcf5a&quot;).
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809897     766 reconciler.go:179] UnmountVolume operation started for volume &quot;kubernetes.io/rbd/6e4e5190-7617-
11e6-b97e-0015c5edcf5a-grafana-cfg&quot; (spec.Name: &quot;grafana-cfg&quot;) from pod &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot; (UID: &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot;).
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809915     766 reconciler.go:179] UnmountVolume operation started for volume &quot;kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data&quot; (spec.Name: &quot;grafana-data&quot;) from pod &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot; (UID: &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot;).
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.840960     766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume &quot;kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06&quot; (OuterVolumeSpecName: &quot;default-token-a7f06&quot;) pod &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot; (UID: &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot;). InnerVolumeSpecName &quot;default-token-a7f06&quot;. PluginName &quot;kubernetes.io/secret&quot;, VolumeGidValue &quot;&quot;
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.843030     766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume &quot;kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data&quot; (OuterVolumeSpecName: &quot;grafana-data&quot;) pod &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot; (UID: &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot;). InnerVolumeSpecName &quot;grafana-data&quot;. PluginName &quot;kubernetes.io/rbd&quot;, VolumeGidValue &quot;&quot;
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.844789     766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume &quot;kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-cfg&quot; (OuterVolumeSpecName: &quot;grafana-cfg&quot;) pod &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot; (UID: &quot;6e4e5190-7617-11e6-b97e-0015c5edcf5a&quot;). InnerVolumeSpecName &quot;grafana-cfg&quot;. PluginName &quot;kubernetes.io/rbd&quot;, VolumeGidValue &quot;&quot;"><pre class="notranslate"><code class="notranslate">Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809860     766 reconciler.go:179] UnmountVolume operation started for volume "kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06" (spec.Name: "default-token-a7f06") from pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-001
5c5edcf5a").
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809897     766 reconciler.go:179] UnmountVolume operation started for volume "kubernetes.io/rbd/6e4e5190-7617-
11e6-b97e-0015c5edcf5a-grafana-cfg" (spec.Name: "grafana-cfg") from pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a").
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.809915     766 reconciler.go:179] UnmountVolume operation started for volume "kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data" (spec.Name: "grafana-data") from pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a").
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.840960     766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4e5190-7617-11e6-b97e-0015c5edcf5a-default-token-a7f06" (OuterVolumeSpecName: "default-token-a7f06") pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a"). InnerVolumeSpecName "default-token-a7f06". PluginName "kubernetes.io/secret", VolumeGidValue ""
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.843030     766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume "kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-data" (OuterVolumeSpecName: "grafana-data") pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a"). InnerVolumeSpecName "grafana-data". PluginName "kubernetes.io/rbd", VolumeGidValue ""
Sep 08 23:03:04 AF002385 docker[630]: I0908 23:03:04.844789     766 operation_executor.go:843] UnmountVolume.TearDown succeeded for volume "kubernetes.io/rbd/6e4e5190-7617-11e6-b97e-0015c5edcf5a-grafana-cfg" (OuterVolumeSpecName: "grafana-cfg") pod "6e4e5190-7617-11e6-b97e-0015c5edcf5a" (UID: "6e4e5190-7617-11e6-b97e-0015c5edcf5a"). InnerVolumeSpecName "grafana-cfg". PluginName "kubernetes.io/rbd", VolumeGidValue ""
</code></pre></div>
<p dir="auto">You can see that each volume appears twice in the output of <code class="notranslate">mount</code> (which is true of apparently everything under <code class="notranslate">/var/lib/kubelet</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)"><pre class="notranslate"><code class="notranslate">/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-cfg type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
/dev/rbd1 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/docker-image-grafana-data type xfs (rw,relatime,seclabel,attr2,inode64,sunit=8192,swidth=8192,noquota)
</code></pre></div>
<p dir="auto">I expect this might be related to recent changes I had to make to how I'm running Docker and Kubelet in order to make secret mounts under Kubernetes 1.3 work. (Bindmounting /var/lib/kubelet onto itself and then marking it shared, per some updated hyperkube instructions... that I can no longer find.) See below for details of how I'm running kubelet.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">That the original node would successfully unmount and unmap the rbd volumes.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Delete a pod with an RBD volume running in a kubelet, possibly with my specific options. (See below.)</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">Here's the systemd unit for running the kubelet:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[Unit]
Description=Kubernetes Worker Node
Wants=network-online.target
After=network-online.target

[Service]
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=-/usr/bin/docker kill kubelet
ExecStartPre=-/usr/bin/docker rm kubelet
ExecStartPre=/usr/bin/bash -c &quot;/usr/bin/mountpoint -q /var/lib/kubelet || /usr/bin/mount --bind /var/lib/kubelet /var/lib/kubelet&quot;
ExecStartPre=/usr/bin/mount --make-shared /var/lib/kubelet
ExecStartPre=/usr/bin/docker pull docker.imvu.com/kubernetes/hyperkube:v1.3.6
ExecStart=/usr/bin/docker run \
        --name kubelet \
        -v /etc/kubernetes:/etc/kubernetes \
        -v /etc/ssl:/etc/ssl:ro \
        -v /etc/ceph:/etc/ceph \
        -v /usr/sbin/modprobe:/sbin/modprobe:ro \
        -v /lib/modules:/lib/modules:ro \
        -v /dev:/dev \
        -v /root/.docker/config.json:/root/.docker/config.json:ro \
        -e HOME=/root/ \
        --net=host \
        --pid=host \
        --privileged \
        --volume=/:/rootfs:ro \
        --volume=/sys:/sys:rw \
        --volume=/var/lib/docker/:/var/lib/docker:rw \
        --volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared \
        --volume=/var/run:/var/run:rw \
        docker.imvu.com/kubernetes/hyperkube:v1.3.6 \
    /hyperkube kubelet \
        --containerized \
        --api-servers=https://10.2.0.24 \
        --register-node=true \
        --allow-privileged=true \
        --config=/etc/kubernetes/manifests \
        --cluster-dns=10.96.0.2 \
        --cluster-domain=k8s.prod.imvu.com \
        --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
        --kube-reserved=cpu=500m,memory=128Mi \
        --node-labels=%s \
        --tls-cert-file=/etc/ssl/host-cert.pem \
        --tls-private-key-file=/etc/ssl/host-key.pem
Restart=always
RestartSec=10"><pre class="notranslate"><code class="notranslate">[Unit]
Description=Kubernetes Worker Node
Wants=network-online.target
After=network-online.target

[Service]
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=-/usr/bin/docker kill kubelet
ExecStartPre=-/usr/bin/docker rm kubelet
ExecStartPre=/usr/bin/bash -c "/usr/bin/mountpoint -q /var/lib/kubelet || /usr/bin/mount --bind /var/lib/kubelet /var/lib/kubelet"
ExecStartPre=/usr/bin/mount --make-shared /var/lib/kubelet
ExecStartPre=/usr/bin/docker pull docker.imvu.com/kubernetes/hyperkube:v1.3.6
ExecStart=/usr/bin/docker run \
        --name kubelet \
        -v /etc/kubernetes:/etc/kubernetes \
        -v /etc/ssl:/etc/ssl:ro \
        -v /etc/ceph:/etc/ceph \
        -v /usr/sbin/modprobe:/sbin/modprobe:ro \
        -v /lib/modules:/lib/modules:ro \
        -v /dev:/dev \
        -v /root/.docker/config.json:/root/.docker/config.json:ro \
        -e HOME=/root/ \
        --net=host \
        --pid=host \
        --privileged \
        --volume=/:/rootfs:ro \
        --volume=/sys:/sys:rw \
        --volume=/var/lib/docker/:/var/lib/docker:rw \
        --volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared \
        --volume=/var/run:/var/run:rw \
        docker.imvu.com/kubernetes/hyperkube:v1.3.6 \
    /hyperkube kubelet \
        --containerized \
        --api-servers=https://10.2.0.24 \
        --register-node=true \
        --allow-privileged=true \
        --config=/etc/kubernetes/manifests \
        --cluster-dns=10.96.0.2 \
        --cluster-domain=k8s.prod.imvu.com \
        --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
        --kube-reserved=cpu=500m,memory=128Mi \
        --node-labels=%s \
        --tls-cert-file=/etc/ssl/host-cert.pem \
        --tls-private-key-file=/etc/ssl/host-key.pem
Restart=always
RestartSec=10
</code></pre></div>
<p dir="auto">(The %s is replaced with a string at generation time.)</p>
<p dir="auto"><code class="notranslate">docker.imvu.com/kubernetes/hyperkube:v1.3.6</code> is a customized image based on the upstream hyperkube Dockerfile, with some extra packages and rebased on top of a version of Ubuntu that we ensure has all security updates. Here's the Dockerfile for it:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="FROM docker.imvu.com/base/ubuntu:16.04

ADD docker/kubernetes/hyperkube/ceph-release.asc /tmp/ceph-release.asc
RUN apt-key add /tmp/ceph-release.asc

ADD docker/kubernetes/hyperkube/ceph.list /etc/apt/sources.list.d/ceph.list
RUN apt-get update

RUN apt-get-for-docker.sh \
    ca-certificates \
    ceph-common \
    conntrack \
    curl \
    ethtool \
    file \
    git \
    iptables \
    socat \
    nfs-common \
    util-linux

RUN cp /usr/bin/nsenter /nsenter

COPY docker/kubernetes/hyperkube/hyperkube /hyperkube
RUN chmod a+rx /hyperkube

RUN ln -s /hyperkube /kubelet"><pre class="notranslate"><code class="notranslate">FROM docker.imvu.com/base/ubuntu:16.04

ADD docker/kubernetes/hyperkube/ceph-release.asc /tmp/ceph-release.asc
RUN apt-key add /tmp/ceph-release.asc

ADD docker/kubernetes/hyperkube/ceph.list /etc/apt/sources.list.d/ceph.list
RUN apt-get update

RUN apt-get-for-docker.sh \
    ca-certificates \
    ceph-common \
    conntrack \
    curl \
    ethtool \
    file \
    git \
    iptables \
    socat \
    nfs-common \
    util-linux

RUN cp /usr/bin/nsenter /nsenter

COPY docker/kubernetes/hyperkube/hyperkube /hyperkube
RUN chmod a+rx /hyperkube

RUN ln -s /hyperkube /kubelet
</code></pre></div>
<p dir="auto">The hyperkube binary there is the one from the 1.3.6 release tarball.</p>
<p dir="auto">The <code class="notranslate">containerPort</code> attribute in the pod manifest could be inferred from the image metadata when not supplied by the developer.</p>
<p dir="auto">That way developers won't have to duplicate the information they already put in EXPOSE in their Dockerfile.</p>
<p dir="auto">This information looks available through the remote API on <code class="notranslate">/image</code> resources under the <code class="notranslate">PortSpecs</code> attribute:<br>
<a href="https://docs.docker.com/reference/api/docker_remote_api_v1.13/#inspect-an-image" rel="nofollow">https://docs.docker.com/reference/api/docker_remote_api_v1.13/#inspect-an-image</a></p>