<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">PetSet, Statefulset, PV, persistent volume, AWS</p>
<p dir="auto">This is similar to issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="188575995" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/36589" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/36589/hovercard" href="https://github.com/kubernetes/kubernetes/issues/36589">#36589</a> except that (a) it's a StatefulSet, and (b) the kubelets were not restarted.</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS, EC2, t2.medium</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
</ul>
<p dir="auto">NAME="CentOS Linux"<br>
VERSION="7 (Core)"<br>
ID="centos"<br>
ID_LIKE="rhel fedora"<br>
VERSION_ID="7"<br>
PRETTY_NAME="CentOS Linux 7 (Core)"<br>
ANSI_COLOR="0;31"<br>
CPE_NAME="cpe:/o:centos:centos:7"<br>
HOME_URL="<a href="https://www.centos.org/" rel="nofollow">https://www.centos.org/</a>"<br>
BUG_REPORT_URL="<a href="https://bugs.centos.org/" rel="nofollow">https://bugs.centos.org/</a>"</p>
<p dir="auto">CENTOS_MANTISBT_PROJECT="CentOS-7"<br>
CENTOS_MANTISBT_PROJECT_VERSION="7"<br>
REDHAT_SUPPORT_PRODUCT="centos"<br>
REDHAT_SUPPORT_PRODUCT_VERSION="7"</p>
<ul dir="auto">
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
</ul>
<p dir="auto">Linux ip-172-31-47-16.us-west-2.compute.internal 3.10.0-327.10.1.el7.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</p>
<ul dir="auto">
<li><strong>Install tools</strong>:  kubeadm</li>
</ul>
<p dir="auto">kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.380+85fe0f1aadf91e", GitCommit:"85fe0f1aadf91e134102cf3c01a9eed11a7e257f", GitTreeState:"clean", BuildDate:"2016-11-02T14:58:17Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}</p>
<ul dir="auto">
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">All EC2 nodes are set up in an IAM role with EBS management access.  To test this, I created a regular PV, which created successfully.</p>
<ol dir="auto">
<li>
<p dir="auto">Built 4-node cluster on EC2 using the Kubeadm instructions for CentOS, including kubeadm init --cloud-provider=aws</p>
</li>
<li>
<p dir="auto">Had to manually fix SSL certs by creating hard links, due to this bug: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="187027417" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/36150" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/36150/hovercard" href="https://github.com/kubernetes/kubernetes/issues/36150">#36150</a></p>
</li>
<li>
<p dir="auto">Added weave network</p>
</li>
<li>
<p dir="auto">Created this StatefulSet: <a href="https://github.com/jberkus/atomicdb/blob/master/patroni_petset/kubernetes/ps-patroni.yaml">https://github.com/jberkus/atomicdb/blob/master/patroni_petset/kubernetes/ps-patroni.yaml</a><br>
(supporting objects are in same dir)</p>
</li>
<li>
<p dir="auto">Service, PVCs, PVs created successfully.  Checking AWS, the EBS volumes were created and are available.</p>
</li>
<li>
<p dir="auto">Pods failed, due to an inability to mount the PVs:</p>
</li>
</ol>
<p dir="auto">FirstSeen	LastSeen	Count	From				SubobjectPath	Type		Reason		Message</p>
<hr>
<p dir="auto">20m		20m		1	{default-scheduler }				Normal		Scheduled	Successfully assigned patroni-0 to ip-172-31-43-85<br>
18m		1m		9	{kubelet ip-172-31-43-85}			Warning		FailedMount	Unable to mount volumes for pod "patroni-0_default(a8e441ac-a854-11e6-bb5c-06027ce5ae69)": timeout expired waiting for volumes to attach/mount for pod "patroni-0"/"default". list of unattached/unmounted volumes=[pgdata]<br>
18m		1m		9	{kubelet ip-172-31-43-85}			Warning		FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "patroni-0"/"default". list of unattached/unmounted volumes=[pgdata]<br>
20m		41s		18	{kubelet ip-172-31-43-85}			Warning		FailedMount	MountVolume.SetUp failed for volume "kubernetes.io/aws-ebs/a8e441ac-a854-11e6-bb5c-06027ce5ae69-pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69" (spec.Name: "pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69") pod "a8e441ac-a854-11e6-bb5c-06027ce5ae69" (UID: "a8e441ac-a854-11e6-bb5c-06027ce5ae69") with: mount failed: exit status 32<br>
Mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-2a/vol-b893ce0c /var/lib/kubelet/pods/a8e441ac-a854-11e6-bb5c-06027ce5ae69/volumes/kubernetes.io~aws-ebs/pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69  [bind]<br>
Output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-2a/vol-b893ce0c does not exist<br>
20m	39s	18	{controller-manager }		Warning	FailedMount	Failed to attach volume "pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69" on node "ip-172-31-43-85" with: error finding instance ip-172-31-43-85: instance not found</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">PVs should have mounted on the pod containers, and the remaining pods should have come up.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">Follow steps above.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<ul dir="auto">
<li>kubectl logs and journalctl -xe don't show any errors related to pvs or mounts</li>
<li>I can't kube exec into the pod because it's down (to check paths)</li>
<li>the same StatefulSet, using only EmptyDir(), works fine.</li>
</ul>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<ul dir="auto">
<li>poddisruptionbudget</li>
<li>pod disruption budget</li>
<li>pdb</li>
</ul>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.2&quot;, GitCommit:&quot;0776eab45fe28f02bbeac0f05ae1a203051a21eb&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-24T22:35:03Z&quot;, GoVersion:&quot;go1.7.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.2&quot;, GitCommit:&quot;0776eab45fe28f02bbeac0f05ae1a203051a21eb&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-24T22:30:23Z&quot;, GoVersion:&quot;go1.7.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:35:03Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:30:23Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">If a <code class="notranslate">policy/v1alpha1</code> PDB still exists after upgrading the master from <code class="notranslate">v1.4.6</code> to <code class="notranslate">v1.5.0-beta.2</code>, <code class="notranslate">kubectl get poddisruptionbudget</code> is broken (even for subsequently-created <code class="notranslate">policy/v1beta1</code> PDBs). The alpha PDB cannot be deleted through <code class="notranslate">kubectl</code> (neither v1.4.6 nor v1.5.0-beta.2), so the user is stuck in this broken state and must roll back the master.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">The existence of the alpha PDB should not break interactions with beta PDBs.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ol dir="auto">
<li>Launch a <code class="notranslate">v1.4.6</code> cluster from an official release build.</li>
<li>Create an alpha PDB:
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="apiVersion: policy/v1alpha1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx
  minAvailable: 4"><pre class="notranslate"><span class="pl-ent">apiVersion</span>: <span class="pl-s">policy/v1alpha1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">PodDisruptionBudget</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">web</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">selector</span>:
    <span class="pl-ent">matchLabels</span>:
      <span class="pl-ent">app</span>: <span class="pl-s">nginx</span>
  <span class="pl-ent">minAvailable</span>: <span class="pl-c1">4</span></pre></div>
</li>
<li>Upgrade the master to <code class="notranslate">v1.5.0-beta.2</code>: <code class="notranslate">cluster/gce/upgrade.sh -M v1.5.0-beta.2</code></li>
<li>Attempt to run <code class="notranslate">kubectl get poddisruptionbudget</code></li>
</ol>
<p dir="auto">With kubectl v1.4.6:</p>
<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ kubectl get poddisruptionbudget
error: group map[certificates.k8s.io:0xc8203eb2d0 componentconfig:0xc8203eb340 extensions:0xc8203b6070 policy:0xc8203b60e0 rbac.authorization.k8s.io:0xc8203b6150 apps:0xc8203eae70 authorization.k8s.io:0xc8203eb030 autoscaling:0xc8203eb0a0 storage.k8s.io:0xc8203b61c0 batch:0xc8203eb260 federation:0xc8203ea930 :0xc8203eae00 authentication.k8s.io:0xc8203eaee0] is already registered"><pre class="notranslate">$ <span class="pl-s1">kubectl get poddisruptionbudget</span>
<span class="pl-c1">error: group map[certificates.k8s.io:0xc8203eb2d0 componentconfig:0xc8203eb340 extensions:0xc8203b6070 policy:0xc8203b60e0 rbac.authorization.k8s.io:0xc8203b6150 apps:0xc8203eae70 authorization.k8s.io:0xc8203eb030 autoscaling:0xc8203eb0a0 storage.k8s.io:0xc8203b61c0 batch:0xc8203eb260 federation:0xc8203ea930 :0xc8203eae00 authentication.k8s.io:0xc8203eaee0] is already registered</span></pre></div>
<p dir="auto">With kubectl v1.5.0-beta.2:</p>
<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ kubectl get poddisruptionbudget
Error from server: no kind &quot;PodDisruptionBudget&quot; is registered for version &quot;policy/v1alpha1&quot;"><pre class="notranslate">$ <span class="pl-s1">kubectl get poddisruptionbudget</span>
<span class="pl-c1">Error from server: no kind "PodDisruptionBudget" is registered for version "policy/v1alpha1"</span></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">If I try to create a beta PDB with the same name, it says:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Error from server (AlreadyExists): error when creating &quot;pdb-beta.yaml&quot;: poddisruptionbudgets.policy &quot;web&quot; already exists"><pre class="notranslate"><code class="notranslate">Error from server (AlreadyExists): error when creating "pdb-beta.yaml": poddisruptionbudgets.policy "web" already exists
</code></pre></div>
<p dir="auto">Attempting to delete the alpha PDB (either by name or by file) fails with:</p>
<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ kubectl delete poddisruptionbudget web
Error from server: no kind &quot;PodDisruptionBudget&quot; is registered for version &quot;policy/v1alpha1&quot;
$ kubectl delete -f pdb-alpha.yaml 
Error from server: error when deleting &quot;pdb-alpha.yaml&quot;: no kind &quot;PodDisruptionBudget&quot; is registered for version &quot;policy/v1alpha1&quot;
$ kubectl delete -f pdb-beta.yaml 
Error from server: error when deleting &quot;pdb-beta.yaml&quot;: no kind &quot;PodDisruptionBudget&quot; is registered for version &quot;policy/v1alpha1&quot;"><pre class="notranslate">$ <span class="pl-s1">kubectl delete poddisruptionbudget web</span>
<span class="pl-c1">Error from server: no kind "PodDisruptionBudget" is registered for version "policy/v1alpha1"</span>
$ <span class="pl-s1">kubectl delete -f pdb-alpha.yaml </span>
<span class="pl-c1">Error from server: error when deleting "pdb-alpha.yaml": no kind "PodDisruptionBudget" is registered for version "policy/v1alpha1"</span>
$ <span class="pl-s1">kubectl delete -f pdb-beta.yaml </span>
<span class="pl-c1">Error from server: error when deleting "pdb-beta.yaml": no kind "PodDisruptionBudget" is registered for version "policy/v1alpha1"</span></pre></div>
<p dir="auto">I can successfully create a beta PDB with a different name and I can specifically <code class="notranslate">kubectl get poddisruptionbudget beta-pdb</code>. However, listing all PDBs (<code class="notranslate">kubectl get poddisruptionbudget</code>) remains broken.</p>