<p dir="auto"><a href="https://github.com/kubernetes/kubernetes/files/583629/kube-error.txt">kube-error.txt</a></p>

<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>1.4.1</strong><br>
<strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
[root@k8s-master ~]# kubectl version<br>
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:13:36Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto">[root@k8s-master ~]# kubectl get nodes<br>
NAME           STATUS    AGE<br>
10.37.116.32   Ready     1h</p>
<p dir="auto">[root@k8s-master ~]# ps -ef|grep kube<br>
root      1767  1708  0 11:32 pts/1    00:00:04 kube-apiserver --logtostderr=true --v=5 --etcd-servers=<a href="https://10.37.116.31:2379,https://10.37.116.32:2379" rel="nofollow">https://10.37.116.31:2379,https://10.37.116.32:2379</a> --etcd-cafile=/ssl/ca.pem --etcd-certfile=/ssl/server.pem --etcd-keyfile=/ssl/server-key.pem --bind-address=10.37.116.31 --secure-port=6443 --advertise-address=10.37.116.31 --allow-privileged=false --service-cluster-ip-range=172.16.0.0/16 --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota --client-ca-file=/ssl/ca.pem --tls-cert-file=/ssl/server.pem --tls-private-key-file=/ssl/server-key.pem<br>
root      1781  1726  0 11:33 pts/2    00:00:01 kube-controller-manager --logtostderr=true --v=5 --master=127.0.0.1:8080 --allocate-node-cidrs=true --cluster-cidr=172.16.0.0/16 --root-ca-file=/ssl/ca.pem --service-account-private-key-file=/ssl/server-key.pem<br>
root      1795  1040  0 11:33 pts/0    00:00:00 kube-scheduler --logtostderr=true --v=5 --master=127.0.0.1:8080</p>
<p dir="auto">[root@k8s-node-1 ~]# ps -ef|grep kube<br>
root     26089  1223  1 11:34 pts/0    00:00:06 kube-proxy --logtostderr=true --v=5 --hostname-override=10.37.116.32 --kubeconfig=/etc/kubernetes/kubecfg --proxy-mode=iptables<br>
root     26529 24882  0 11:34 pts/1    00:00:04 kubelet --logtostderr=true --v=5 --address=10.37.116.32 --port=10250 --hostname-override=10.37.116.32 --kubeconfig=/etc/kubernetes/kubecfg --require-kubeconfig=true --pod-infra-container-image=10.37.116.11:8443/pause:latest --allow-privileged=false --tls-cert-file=/ssl/server.pem --tls-private-key-file=/ssl/server-key.pem</p>
<p dir="auto">[root@k8s-node-1 ~]# cat /etc/kubernetes/kubecfg<br>
apiVersion: v1<br>
kind: Config<br>
users:</p>
<ul dir="auto">
<li>name: kubelet<br>
user:<br>
client-certificate: /ssl/server.pem<br>
client-key: /ssl/server-key.pem<br>
clusters:</li>
<li>name: dev<br>
cluster:<br>
certificate-authority: /ssl/ca.pem<br>
server: <a href="https://10.37.116.31:6443" rel="nofollow">https://10.37.116.31:6443</a><br>
contexts:</li>
<li>context:<br>
cluster: dev<br>
user: kubelet<br>
name: service-account-context<br>
current-context: service-account-context</li>
</ul>
<p dir="auto">[root@k8s-master ~]# kubectl get sa --all-namespaces -o yaml<br>
apiVersion: v1<br>
items:</p>
<ul dir="auto">
<li>apiVersion: v1<br>
kind: ServiceAccount<br>
metadata:<br>
creationTimestamp: 2016-11-10T03:24:12Z<br>
name: default<br>
namespace: default<br>
resourceVersion: "2732"<br>
selfLink: /api/v1/namespaces/default/serviceaccounts/default<br>
uid: 26a7b030-a6f5-11e6-84f0-080027acd9e1<br>
secrets:
<ul dir="auto">
<li>name: default-token-g9xgv</li>
</ul>
</li>
<li>apiVersion: v1<br>
kind: ServiceAccount<br>
metadata:<br>
creationTimestamp: 2016-11-10T03:24:26Z<br>
name: default<br>
namespace: kube-system<br>
resourceVersion: "2754"<br>
selfLink: /api/v1/namespaces/kube-system/serviceaccounts/default<br>
uid: 2edaf5ed-a6f5-11e6-84f0-080027acd9e1<br>
secrets:
<ul dir="auto">
<li>name: default-token-rsqoz<br>
kind: List<br>
metadata: {}</li>
</ul>
</li>
</ul>
<p dir="auto">[root@k8s-master ~]# kubectl get secrets --all-namespaces -o yaml<br>
apiVersion: v1<br>
items:</p>
<ul dir="auto">
<li>apiVersion: v1<br>
data:<br>
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVY1puRlRXd29UMWxpeXVtT3JGYXcxaVBzQTVrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RGpBTUJnTlZCQWdUQlVoMWJtRnVNUkV3RHdZRFZRUUhFd2hEYUdGdQpaM05vWVRFU01CQUdBMVVFQ2hNSlN6aHpMQ0JKYm1NdU1SRXdEd1lEVlFRTEV3aE5hVzRnVkdWamFERU1NQW9HCkExVUVBeE1EU3poVE1CNFhEVEUyTVRFeE1EQXlNemt3TUZvWERUSXhNVEV3T1RBeU16a3dNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RGpBTUJnTlZCQWdUQlVoMWJtRnVNUkV3RHdZRFZRUUhFd2hEYUdGdVozTm9ZVEVTTUJBRwpBMVVFQ2hNSlN6aHpMQ0JKYm1NdU1SRXdEd1lEVlFRTEV3aE5hVzRnVkdWamFERU1NQW9HQTFVRUF4TURTemhUCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBd1c1R0dXR0JCR0ZMUG53cHVraFoKYk4zeEV2cUQzWkdkWHZWNzI2NTRSeHM1VGREcWoxZXN6ZFgvVUxpd3VEZEQ5cFNpYWlVb1IzMytCZ1dFRHRGSApoRmlMUzhBdW5maXkrd0xlam0wVktrdHRhcTA2bjd0RDNyWll0cGc1c1BCNlhHVGtqVHlsZWdpNFhCRE55ajUvCmtyRjliVXpmZmI5VVF2TGhVYUpveWIwQWsvRXNFdUQ2Unc5OVpFempGQS9jbHdMWFA5MmowTlk1MEc5N3ZwN0wKMVNsZ1ZudHdveUNYaE1ua2R3LysyMVdkQVprdUtjbERSSUY2eW45bFU0Y05ZOE96ZVlIQ3lnazFqMVN5VHRhNgpRUCsxZDI2QXRnaHpGZkhzRGxidkFyUmt1ei85RWhxRmRoenVuUm8wSWVFdXQvWkx3OGZHQ2MzSGY3NmgyS2JlClhRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVVUaXoxekdYR1ZZMXpIT3dZNHZDOXhuc2RxdFV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUpNZwp1ZVg3V0thRllGc2srUmVYaitjRXp3aStUZjBiZ09abFVaVVJVaUlvWTc1NlA3WW9ZUkZUNHJZL1kyb1BUUHBQCjdIOUVLdGkvRzFFTE5hbVBCZ3ovejV2aDJTVzlRZ25ZcGVjeGpDM0tqODFKWVlWOTV6NEJLbE9Tb2JEMlpiYnEKb25KL1NPK3ZSWGhraGI2c3VpdHdvNy9WNGlWWG54MUdNTU5NVldKUWFDYWpZaEx6ZWZIemdQNG0wamNEUnBudwp3d09UZklSYVdSRUtGT3JvcTlBTFdSdFBBcEFTV0c4d2ZINVVMYk40NzczbHJvZ0V4d1ltVnVqNDYwY1hhVzdwCkVvZHdWVG04dXE2MzNsQ3RNS0YyVVpCbzZ5V2tMellBclU1NHk5MmZSMmdiampLckJhcGVKYmZwN1NFUmhZNHIKb0dsOWdpWFZLZlVRMVhxUzZlZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=<br>
namespace: ZGVmYXVsdA==<br>
token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbVJsWm1GMWJIUXRkRzlyWlc0dFp6bDRaM1lpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWkdWbVlYVnNkQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJakkyWVRkaU1ETXdMV0UyWmpVdE1URmxOaTA0TkdZd0xUQTRNREF5TjJGalpEbGxNU0lzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwa1pXWmhkV3gwT21SbFptRjFiSFFpZlEuYU1kS0RfN2lVVXZsVG5LSVNUTFRqVk5QcTE4TEktSGVPMnR0dlhJZHlQSVFFaTNrQ3dtYnh2ZVJCbEdPbjBkN3lEeUxYX09ZZldmSm9VcE1sYVRfeV9XcFRWMml0dUlRbUVfd3N0X05iOW91ckVRZEFHbmJnS0RwX281eWJlS09pU2UzN0h6RFE3ekVHNHh4QWdxZTR1M1NTNl9PQmM0OHdsTWtsUF9LSW4tSTA2c3EzMXRwTV9TMUY1VFpsVnA2U0dnbXU5WWF0TVpySmx1VXEzMllaazQ2R0RtVFdGWW05WERXU2RlQlVPbkxWWnE4UEREZjE4TGJJZVhwUVZ5NWUyem1HRm0tem9uU0xQOV9DclUzeTBCRGRJb1lnWlJvcFNFUkNJSVRyN1Rlay1PSVJwWU91bjZSR3dTNVhtb0ozZzg5a0hHSUpTRGNoUldaOGl5cGJB<br>
kind: Secret<br>
metadata:<br>
annotations:<br>
kubernetes.io/service-account.name: default<br>
kubernetes.io/service-account.uid: 26a7b030-a6f5-11e6-84f0-080027acd9e1<br>
creationTimestamp: 2016-11-10T03:24:12Z<br>
name: default-token-g9xgv<br>
namespace: default<br>
resourceVersion: "2731"<br>
selfLink: /api/v1/namespaces/default/secrets/default-token-g9xgv<br>
uid: 26af0a29-a6f5-11e6-84f0-080027acd9e1<br>
type: kubernetes.io/service-account-token</li>
<li>apiVersion: v1<br>
data:<br>
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURtakNDQW9LZ0F3SUJBZ0lVY1puRlRXd29UMWxpeXVtT3JGYXcxaVBzQTVrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RGpBTUJnTlZCQWdUQlVoMWJtRnVNUkV3RHdZRFZRUUhFd2hEYUdGdQpaM05vWVRFU01CQUdBMVVFQ2hNSlN6aHpMQ0JKYm1NdU1SRXdEd1lEVlFRTEV3aE5hVzRnVkdWamFERU1NQW9HCkExVUVBeE1EU3poVE1CNFhEVEUyTVRFeE1EQXlNemt3TUZvWERUSXhNVEV3T1RBeU16a3dNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RGpBTUJnTlZCQWdUQlVoMWJtRnVNUkV3RHdZRFZRUUhFd2hEYUdGdVozTm9ZVEVTTUJBRwpBMVVFQ2hNSlN6aHpMQ0JKYm1NdU1SRXdEd1lEVlFRTEV3aE5hVzRnVkdWamFERU1NQW9HQTFVRUF4TURTemhUCk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBd1c1R0dXR0JCR0ZMUG53cHVraFoKYk4zeEV2cUQzWkdkWHZWNzI2NTRSeHM1VGREcWoxZXN6ZFgvVUxpd3VEZEQ5cFNpYWlVb1IzMytCZ1dFRHRGSApoRmlMUzhBdW5maXkrd0xlam0wVktrdHRhcTA2bjd0RDNyWll0cGc1c1BCNlhHVGtqVHlsZWdpNFhCRE55ajUvCmtyRjliVXpmZmI5VVF2TGhVYUpveWIwQWsvRXNFdUQ2Unc5OVpFempGQS9jbHdMWFA5MmowTlk1MEc5N3ZwN0wKMVNsZ1ZudHdveUNYaE1ua2R3LysyMVdkQVprdUtjbERSSUY2eW45bFU0Y05ZOE96ZVlIQ3lnazFqMVN5VHRhNgpRUCsxZDI2QXRnaHpGZkhzRGxidkFyUmt1ei85RWhxRmRoenVuUm8wSWVFdXQvWkx3OGZHQ2MzSGY3NmgyS2JlClhRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlYKSFE0RUZnUVVUaXoxekdYR1ZZMXpIT3dZNHZDOXhuc2RxdFV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUpNZwp1ZVg3V0thRllGc2srUmVYaitjRXp3aStUZjBiZ09abFVaVVJVaUlvWTc1NlA3WW9ZUkZUNHJZL1kyb1BUUHBQCjdIOUVLdGkvRzFFTE5hbVBCZ3ovejV2aDJTVzlRZ25ZcGVjeGpDM0tqODFKWVlWOTV6NEJLbE9Tb2JEMlpiYnEKb25KL1NPK3ZSWGhraGI2c3VpdHdvNy9WNGlWWG54MUdNTU5NVldKUWFDYWpZaEx6ZWZIemdQNG0wamNEUnBudwp3d09UZklSYVdSRUtGT3JvcTlBTFdSdFBBcEFTV0c4d2ZINVVMYk40NzczbHJvZ0V4d1ltVnVqNDYwY1hhVzdwCkVvZHdWVG04dXE2MzNsQ3RNS0YyVVpCbzZ5V2tMellBclU1NHk5MmZSMmdiampLckJhcGVKYmZwN1NFUmhZNHIKb0dsOWdpWFZLZlVRMVhxUzZlZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=<br>
namespace: a3ViZS1zeXN0ZW0=<br>
token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJblI1Y0NJNklrcFhWQ0o5LmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpyZFdKbExYTjVjM1JsYlNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZqY21WMExtNWhiV1VpT2lKa1pXWmhkV3gwTFhSdmEyVnVMWEp6Y1c5Nklpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXVibUZ0WlNJNkltUmxabUYxYkhRaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sY25acFkyVXRZV05qYjNWdWRDNTFhV1FpT2lJeVpXUmhaalZsWkMxaE5tWTFMVEV4WlRZdE9EUm1NQzB3T0RBd01qZGhZMlE1WlRFaUxDSnpkV0lpT2lKemVYTjBaVzA2YzJWeWRtbGpaV0ZqWTI5MWJuUTZhM1ZpWlMxemVYTjBaVzA2WkdWbVlYVnNkQ0o5LlRBSTdoNVBLLWRYX1QyN0poaGkzblpsaEMwYjNfT21TWHI1ZUw4b0RlbVZhMHk5ZHotQ3NqVlVZam9YMkJzeFRuc0QwU2JHc1YwM1NEVHJOakVJZGZRNDg4NGVRejJGRkxvckVJQkJ6N1hDM3d1RkNMMnNkcExhcnltY2U2cFBRX1hFVDR3TzZpYjVLRHBJODhkUmxMOUt4QzNoUG5UTDh6VkEyU3hkdUU2MU9qZlQ4NGM3MkdKMTh4cXdzRmZnWFhwaEVvODVkV19Tb3Qza0l0b2hEMEtFcDVNLUoxc3dlMm5Sdk1DRkxBcHpYaC1PaUU0QlA1RkNUTEFocE0xMnlrUUNqLTZ0Y1dkMmNpanVKWjZHSXVrOTZFRHlTNWdxZXJBOHVEdWV4X3lLdnBiMUctSHFyVlBpRmZXRnM5Wk1yLXBVeVoyOG1JV01NcHFKTzdiX0JZUQ==<br>
kind: Secret<br>
metadata:<br>
annotations:<br>
kubernetes.io/service-account.name: default<br>
kubernetes.io/service-account.uid: 2edaf5ed-a6f5-11e6-84f0-080027acd9e1<br>
creationTimestamp: 2016-11-10T03:24:26Z<br>
name: default-token-rsqoz<br>
namespace: kube-system<br>
resourceVersion: "2753"<br>
selfLink: /api/v1/namespaces/kube-system/secrets/default-token-rsqoz<br>
uid: 2edf140b-a6f5-11e6-84f0-080027acd9e1<br>
type: kubernetes.io/service-account-token<br>
kind: List<br>
metadata: {}</li>
</ul>
<p dir="auto">[root@k8s-master ~]# kubectl get svc,po --all-namespaces<br>
NAMESPACE     NAME                       CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE<br>
default       svc/kubernetes             172.16.0.1              443/TCP   54m<br>
kube-system   svc/kubernetes-dashboard   172.16.58.226           80/TCP    45m<br>
NAMESPACE     NAME                                   READY     STATUS             RESTARTS   AGE<br>
kube-system   po/kubernetes-dashboard-v1.4.1-6s1qb   0/1       CrashLoopBackOff   6          7m</p>
<p dir="auto">[root@k8s-master ~]# kubectl -n kube-system get po kubernetes-dashboard-v1.4.1-6s1qb -o yaml<br>
apiVersion: v1<br>
kind: Pod<br>
metadata:<br>
annotations:<br>
kubernetes.io/created-by: |<br>
{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"kube-system","name":"kubernetes-dashboard-v1.4.1","uid":"deb11313-a6f1-11e6-84f0-080027acd9e1","apiVersion":"v1","resourceVersion":"3214"}}<br>
creationTimestamp: 2016-11-10T03:38:02Z<br>
generateName: kubernetes-dashboard-v1.4.1-<br>
labels:<br>
k8s-app: kubernetes-dashboard<br>
kubernetes.io/cluster-service: "true"<br>
version: v1.4.1<br>
name: kubernetes-dashboard-v1.4.1-6s1qb<br>
namespace: kube-system<br>
ownerReferences:</p>
<ul dir="auto">
<li>apiVersion: v1<br>
controller: true<br>
kind: ReplicationController<br>
name: kubernetes-dashboard-v1.4.1<br>
uid: deb11313-a6f1-11e6-84f0-080027acd9e1<br>
resourceVersion: "4813"<br>
selfLink: /api/v1/namespaces/kube-system/pods/kubernetes-dashboard-v1.4.1-6s1qb<br>
uid: 1513fc5e-a6f7-11e6-b259-080027acd9e1<br>
spec:<br>
containers:</li>
<li>args:
<ul dir="auto">
<li>--apiserver-host=<a href="https://10.37.116.31:6443" rel="nofollow">https://10.37.116.31:6443</a><br>
env:</li>
<li>name: KUBECONFIG<br>
value: /etc/kubernetes/kubecfg<br>
image: 10.37.116.11:8443/kubernetes-dashboard-amd64:v1.4.1<br>
imagePullPolicy: IfNotPresent<br>
livenessProbe:<br>
failureThreshold: 3<br>
httpGet:<br>
path: /<br>
port: 9090<br>
scheme: HTTP<br>
initialDelaySeconds: 30<br>
periodSeconds: 10<br>
successThreshold: 1<br>
timeoutSeconds: 30<br>
name: kubernetes-dashboard<br>
ports:</li>
<li>containerPort: 9090<br>
protocol: TCP<br>
resources:<br>
limits:<br>
cpu: 100m<br>
memory: 50Mi<br>
requests:<br>
cpu: 100m<br>
memory: 50Mi<br>
terminationMessagePath: /dev/termination-log<br>
volumeMounts:</li>
<li>mountPath: /etc/pki<br>
name: etcpki<br>
readOnly: true</li>
<li>mountPath: /etc/kubernetes<br>
name: config<br>
readOnly: true</li>
<li>mountPath: /var/run/secrets/kubernetes.io/serviceaccount<br>
name: default-token-rsqoz<br>
readOnly: true<br>
dnsPolicy: ClusterFirst<br>
nodeName: 10.37.116.32<br>
restartPolicy: Always<br>
securityContext: {}<br>
serviceAccount: default<br>
serviceAccountName: default<br>
terminationGracePeriodSeconds: 30<br>
volumes:</li>
</ul>
</li>
<li>hostPath:<br>
path: /etc/pki<br>
name: etcpki</li>
<li>hostPath:<br>
path: /etc/kubernetes<br>
name: config</li>
<li>name: default-token-rsqoz<br>
secret:<br>
defaultMode: 420<br>
secretName: default-token-rsqoz<br>
status:<br>
conditions:</li>
<li>lastProbeTime: null<br>
lastTransitionTime: 2016-11-10T03:38:02Z<br>
status: "True"<br>
type: Initialized</li>
<li>lastProbeTime: null<br>
lastTransitionTime: 2016-11-10T03:43:45Z<br>
message: 'containers with unready status: [kubernetes-dashboard]'<br>
reason: ContainersNotReady<br>
status: "False"<br>
type: Ready</li>
<li>lastProbeTime: null<br>
lastTransitionTime: 2016-11-10T03:38:02Z<br>
status: "True"<br>
type: PodScheduled<br>
containerStatuses:</li>
<li>containerID: docker://c196c881084ada28a001b207879e1317c0fd9aa01fc8343ee6ce7f8a42b35666<br>
image: 10.37.116.11:8443/kubernetes-dashboard-amd64:v1.4.1<br>
imageID: docker://sha256:1dda73f463b239955d4cf94a9cd525ab5306bee0eb53c17534d3282bc50ae7aa<br>
lastState:<br>
terminated:<br>
containerID: docker://c196c881084ada28a001b207879e1317c0fd9aa01fc8343ee6ce7f8a42b35666<br>
exitCode: 1<br>
finishedAt: 2016-11-10T03:48:49Z<br>
reason: Error<br>
startedAt: 2016-11-10T03:48:48Z<br>
name: kubernetes-dashboard<br>
ready: false<br>
restartCount: 7<br>
state:<br>
waiting:<br>
message: Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-v1.4.1-6s1qb_kube-system(1513fc5e-a6f7-11e6-b259-080027acd9e1)<br>
reason: CrashLoopBackOff<br>
hostIP: 10.37.116.32<br>
phase: Running<br>
podIP: 172.16.22.130<br>
startTime: 2016-11-10T03:38:02Z</li>
</ul>
<p dir="auto">[root@k8s-master ~]# kubectl -n kube-system logs po/kubernetes-dashboard-v1.4.1-6s1qb<br>
Using HTTP port: 9090<br>
Using apiserver-host location: <a href="https://10.37.116.31:6443" rel="nofollow">https://10.37.116.31:6443</a><br>
Creating API server client for <a href="https://10.37.116.31:6443" rel="nofollow">https://10.37.116.31:6443</a><br>
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: the server has asked for the client to provide credentials</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">No.</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<ul dir="auto">
<li>upgrade volume</li>
<li>mount timeout</li>
<li>ContainerCreating</li>
</ul>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;5+&quot;, GitVersion:&quot;v1.5.0-beta.2&quot;, GitCommit:&quot;0776eab45fe28f02bbeac0f05ae1a203051a21eb&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-11-24T22:30:23Z&quot;, GoVersion:&quot;go1.7.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"0776eab45fe28f02bbeac0f05ae1a203051a21eb", GitTreeState:"clean", BuildDate:"2016-11-24T22:30:23Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):
<ul dir="auto">
<li>master:
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="BUILD_ID=8977.0.0
NAME=&quot;Container-VM Image&quot;
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=56
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME=&quot;Google Container-VM Image&quot;
VERSION=56
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL=&quot;https://cloud.google.com/compute/docs/containers/vm-image/&quot;
ID=gci"><pre class="notranslate"><code class="notranslate">BUILD_ID=8977.0.0
NAME="Container-VM Image"
GOOGLE_CRASH_ID=Lakitu
VERSION_ID=56
BUG_REPORT_URL=https://crbug.com/new
PRETTY_NAME="Google Container-VM Image"
VERSION=56
GOOGLE_METRICS_PRODUCT_ID=26
HOME_URL="https://cloud.google.com/compute/docs/containers/vm-image/"
ID=gci
</code></pre></div>
</li>
<li>minion:
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PRETTY_NAME=&quot;Debian GNU/Linux 7 (wheezy)&quot;
NAME=&quot;Debian GNU/Linux&quot;
VERSION_ID=&quot;7&quot;
VERSION=&quot;7 (wheezy)&quot;
ID=debian
ANSI_COLOR=&quot;1;31&quot;
HOME_URL=&quot;http://www.debian.org/&quot;
SUPPORT_URL=&quot;http://www.debian.org/support/&quot;
BUG_REPORT_URL=&quot;http://bugs.debian.org/&quot;"><pre class="notranslate"><code class="notranslate">PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
NAME="Debian GNU/Linux"
VERSION_ID="7"
VERSION="7 (wheezy)"
ID=debian
ANSI_COLOR="1;31"
HOME_URL="http://www.debian.org/"
SUPPORT_URL="http://www.debian.org/support/"
BUG_REPORT_URL="http://bugs.debian.org/"
</code></pre></div>
</li>
</ul>
</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):
<ul dir="auto">
<li>master: <code class="notranslate">Linux kubernetes-master 4.4.21+ #1 SMP Thu Nov 10 02:50:15 PST 2016 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux</code></li>
<li>minion: <code class="notranslate">Linux kubernetes-minion-group-4klk 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux</code></li>
</ul>
</li>
<li><strong>Install tools</strong>: <code class="notranslate">cluster/gce/upgrade.sh</code></li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Pods with PVCs get stuck in ContainerCreating after Node gets upgraded. The error is <code class="notranslate">timeout expired waiting for volumes to attach/mount</code> as shown below:</p>
<div class="highlight highlight-text-shell-session notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ kubectl describe pod web-1
Name:           web-1
Namespace:      default
Node:           kubernetes-minion-group-54hw/10.128.0.4
Start Time:     Mon, 28 Nov 2016 11:52:11 -0800
Labels:         app=nginx
Status:         Pending
IP:
Controllers:    PetSet/web
Containers:
  nginx:
    Container ID:
    Image:              gcr.io/google_containers/nginx-slim:0.8
    Image ID:   
    Port:               80/TCP
    Requests:
      cpu:              100m
    State:              Waiting
      Reason:           ContainerCreating
    Ready:              False
    Restart Count:      0
    Volume Mounts:
      /usr/share/nginx/html from www (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xim26 (ro)
    Environment Variables:      &lt;none&gt;
Conditions:
  Type          Status
  Initialized   True 
  Ready         False 
  PodScheduled  True 
Volumes:
  www:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  www-web-1
    ReadOnly:   false
  default-token-xim26:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-xim26
QoS Class:      Burstable
Tolerations:    &lt;none&gt;
Events:
  FirstSeen     LastSeen        Count   From                                    SubObjectPath       Type             Reason          Message
  ---------     --------        -----   ----                                    -------------       -------- ------          -------
  40m           40m             1       {default-scheduler }                                        Normal           Scheduled       Successfully assigned web-1 to kubernetes-minion-group-54hw
  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Pulling         pulling image &quot;gcr.io/google_containers/nginx-slim:0.8&quot;
  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Pulled          Successfully pulled image &quot;gcr.io/google_containers/nginx-slim:0.8&quot;
  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Created         Created container with docker id d2d15140a74a; Security:[seccomp=unconfined]
  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Started         Started container with docker id d2d15140a74a
  16m           16m             4       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     MountVolume.SetUp failed for volume &quot;kubernetes.io/secret/270e4f19-b5a4-11e6-8922-42010a800002-default-token-xim26&quot; (spec.Name: &quot;default-token-xim26&quot;) pod &quot;270e4f19-b5a4-11e6-8922-42010a800002&quot; (UID: &quot;270e4f19-b5a4-11e6-8922-42010a800002&quot;) with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-xim26: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host
  16m           16m             1       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     MountVolume.SetUp failed for volume &quot;kubernetes.io/secret/270e4f19-b5a4-11e6-8922-42010a800002-default-token-xim26&quot; (spec.Name: &quot;default-token-xim26&quot;) pod &quot;270e4f19-b5a4-11e6-8922-42010a800002&quot; (UID: &quot;270e4f19-b5a4-11e6-8922-42010a800002&quot;) with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-xim26: dial tcp 10.128.0.2:443: i/o timeout
  15m           15m             3       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     MountVolume.SetUp failed for volume &quot;kubernetes.io/secret/270e4f19-b5a4-11e6-8922-42010a800002-default-token-xim26&quot; (spec.Name: &quot;default-token-xim26&quot;) pod &quot;270e4f19-b5a4-11e6-8922-42010a800002&quot; (UID: &quot;270e4f19-b5a4-11e6-8922-42010a800002&quot;) with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-xim26: dial tcp 10.128.0.2:443: getsockopt: connection refused
  9m            9m              1       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedSync      Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR]
  9m            9m              3       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedSync      Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR]
  6m            14s             4       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     Unable to mount volumes for pod &quot;web-1_default(270e4f19-b5a4-11e6-8922-42010a800002)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;web-1&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[www]
  6m            14s             4       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;web-1&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[www]"><pre class="notranslate">$ <span class="pl-s1">kubectl describe pod web-1</span>
<span class="pl-c1">Name:           web-1</span>
<span class="pl-c1">Namespace:      default</span>
<span class="pl-c1">Node:           kubernetes-minion-group-54hw/10.128.0.4</span>
<span class="pl-c1">Start Time:     Mon, 28 Nov 2016 11:52:11 -0800</span>
<span class="pl-c1">Labels:         app=nginx</span>
<span class="pl-c1">Status:         Pending</span>
<span class="pl-c1">IP:</span>
<span class="pl-c1">Controllers:    PetSet/web</span>
<span class="pl-c1">Containers:</span>
<span class="pl-c1">  nginx:</span>
<span class="pl-c1">    Container ID:</span>
<span class="pl-c1">    Image:              gcr.io/google_containers/nginx-slim:0.8</span>
<span class="pl-c1">    Image ID:   </span>
<span class="pl-c1">    Port:               80/TCP</span>
<span class="pl-c1">    Requests:</span>
<span class="pl-c1">      cpu:              100m</span>
<span class="pl-c1">    State:              Waiting</span>
<span class="pl-c1">      Reason:           ContainerCreating</span>
<span class="pl-c1">    Ready:              False</span>
<span class="pl-c1">    Restart Count:      0</span>
<span class="pl-c1">    Volume Mounts:</span>
<span class="pl-c1">      /usr/share/nginx/html from www (rw)</span>
<span class="pl-c1">      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xim26 (ro)</span>
<span class="pl-c1">    Environment Variables:      &lt;none&gt;</span>
<span class="pl-c1">Conditions:</span>
<span class="pl-c1">  Type          Status</span>
<span class="pl-c1">  Initialized   True </span>
<span class="pl-c1">  Ready         False </span>
<span class="pl-c1">  PodScheduled  True </span>
<span class="pl-c1">Volumes:</span>
<span class="pl-c1">  www:</span>
<span class="pl-c1">    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)</span>
<span class="pl-c1">    ClaimName:  www-web-1</span>
<span class="pl-c1">    ReadOnly:   false</span>
<span class="pl-c1">  default-token-xim26:</span>
<span class="pl-c1">    Type:       Secret (a volume populated by a Secret)</span>
<span class="pl-c1">    SecretName: default-token-xim26</span>
<span class="pl-c1">QoS Class:      Burstable</span>
<span class="pl-c1">Tolerations:    &lt;none&gt;</span>
<span class="pl-c1">Events:</span>
<span class="pl-c1">  FirstSeen     LastSeen        Count   From                                    SubObjectPath       Type             Reason          Message</span>
<span class="pl-c1">  ---------     --------        -----   ----                                    -------------       -------- ------          -------</span>
<span class="pl-c1">  40m           40m             1       {default-scheduler }                                        Normal           Scheduled       Successfully assigned web-1 to kubernetes-minion-group-54hw</span>
<span class="pl-c1">  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Pulling         pulling image "gcr.io/google_containers/nginx-slim:0.8"</span>
<span class="pl-c1">  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Pulled          Successfully pulled image "gcr.io/google_containers/nginx-slim:0.8"</span>
<span class="pl-c1">  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Created         Created container with docker id d2d15140a74a; Security:[seccomp=unconfined]</span>
<span class="pl-c1">  40m           40m             1       {kubelet kubernetes-minion-group-54hw}  spec.containers{nginx}       Normal          Started         Started container with docker id d2d15140a74a</span>
<span class="pl-c1">  16m           16m             4       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     MountVolume.SetUp failed for volume "kubernetes.io/secret/270e4f19-b5a4-11e6-8922-42010a800002-default-token-xim26" (spec.Name: "default-token-xim26") pod "270e4f19-b5a4-11e6-8922-42010a800002" (UID: "270e4f19-b5a4-11e6-8922-42010a800002") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-xim26: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host</span>
<span class="pl-c1">  16m           16m             1       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     MountVolume.SetUp failed for volume "kubernetes.io/secret/270e4f19-b5a4-11e6-8922-42010a800002-default-token-xim26" (spec.Name: "default-token-xim26") pod "270e4f19-b5a4-11e6-8922-42010a800002" (UID: "270e4f19-b5a4-11e6-8922-42010a800002") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-xim26: dial tcp 10.128.0.2:443: i/o timeout</span>
<span class="pl-c1">  15m           15m             3       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     MountVolume.SetUp failed for volume "kubernetes.io/secret/270e4f19-b5a4-11e6-8922-42010a800002-default-token-xim26" (spec.Name: "default-token-xim26") pod "270e4f19-b5a4-11e6-8922-42010a800002" (UID: "270e4f19-b5a4-11e6-8922-42010a800002") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-xim26: dial tcp 10.128.0.2:443: getsockopt: connection refused</span>
<span class="pl-c1">  9m            9m              1       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedSync      Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR]</span>
<span class="pl-c1">  9m            9m              3       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedSync      Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR]</span>
<span class="pl-c1">  6m            14s             4       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedMount     Unable to mount volumes for pod "web-1_default(270e4f19-b5a4-11e6-8922-42010a800002)": timeout expired waiting for volumes to attach/mount for pod "web-1"/"default". list of unattached/unmounted volumes=[www]</span>
<span class="pl-c1">  6m            14s             4       {kubelet kubernetes-minion-group-54hw}                      Warning          FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "web-1"/"default". list of unattached/unmounted volumes=[www]</span></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Upon Node upgrade, Pods should either be terminated (so a controller can recreate them) or they should come back to Running on their own.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ol dir="auto">
<li>
<p dir="auto">Start a v1.4 cluster on GCE.</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="git checkout release-1.4
git pull upstream release-1.4
make clean quick-release
cluster/kube-up.sh"><pre class="notranslate">git checkout release-1.4
git pull upstream release-1.4
make clean quick-release
cluster/kube-up.sh</pre></div>
</li>
<li>
<p dir="auto">Create the following example PetSet and wait for all Pods to be Running.</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="kubectl create -f petset.yaml"><pre class="notranslate">kubectl create -f petset.yaml</pre></div>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  # *.nginx.default.svc.cluster.local
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1alpha1
kind: PetSet
metadata:
  name: web
spec:
  serviceName: &quot;nginx&quot;
  replicas: 5
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        pod.alpha.kubernetes.io/initialized: &quot;true&quot;
    spec:
      containers:
      - name: nginx
        image: gcr.io/google_containers/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ &quot;ReadWriteOnce&quot; ]
      resources:
        requests:
          storage: 1Gi"><pre class="notranslate"><span class="pl-c"><span class="pl-c">#</span> A headless service to create DNS records</span>
<span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">Service</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
  <span class="pl-ent">labels</span>:
    <span class="pl-ent">app</span>: <span class="pl-s">nginx</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">ports</span>:
  - <span class="pl-ent">port</span>: <span class="pl-c1">80</span>
    <span class="pl-ent">name</span>: <span class="pl-s">web</span>
  <span class="pl-c"><span class="pl-c">#</span> *.nginx.default.svc.cluster.local</span>
  <span class="pl-ent">clusterIP</span>: <span class="pl-s">None</span>
  <span class="pl-ent">selector</span>:
    <span class="pl-ent">app</span>: <span class="pl-s">nginx</span>
---
<span class="pl-ent">apiVersion</span>: <span class="pl-s">apps/v1alpha1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">PetSet</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">web</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">serviceName</span>: <span class="pl-s"><span class="pl-pds">"</span>nginx<span class="pl-pds">"</span></span>
  <span class="pl-ent">replicas</span>: <span class="pl-c1">5</span>
  <span class="pl-ent">template</span>:
    <span class="pl-ent">metadata</span>:
      <span class="pl-ent">labels</span>:
        <span class="pl-ent">app</span>: <span class="pl-s">nginx</span>
      <span class="pl-ent">annotations</span>:
        <span class="pl-ent">pod.alpha.kubernetes.io/initialized</span>: <span class="pl-s"><span class="pl-pds">"</span>true<span class="pl-pds">"</span></span>
    <span class="pl-ent">spec</span>:
      <span class="pl-ent">containers</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">nginx</span>
        <span class="pl-ent">image</span>: <span class="pl-s">gcr.io/google_containers/nginx-slim:0.8</span>
        <span class="pl-ent">ports</span>:
        - <span class="pl-ent">containerPort</span>: <span class="pl-c1">80</span>
          <span class="pl-ent">name</span>: <span class="pl-s">web</span>
        <span class="pl-ent">volumeMounts</span>:
        - <span class="pl-ent">name</span>: <span class="pl-s">www</span>
          <span class="pl-ent">mountPath</span>: <span class="pl-s">/usr/share/nginx/html</span>
  <span class="pl-ent">volumeClaimTemplates</span>:
  - <span class="pl-ent">metadata</span>:
      <span class="pl-ent">name</span>: <span class="pl-s">www</span>
      <span class="pl-ent">annotations</span>:
        <span class="pl-ent">volume.alpha.kubernetes.io/storage-class</span>: <span class="pl-s">anything</span>
    <span class="pl-ent">spec</span>:
      <span class="pl-ent">accessModes</span>: <span class="pl-s">[ "ReadWriteOnce" ]</span>
      <span class="pl-ent">resources</span>:
        <span class="pl-ent">requests</span>:
          <span class="pl-ent">storage</span>: <span class="pl-s">1Gi</span></pre></div>
</li>
<li>
<p dir="auto">Delete PetSet without cascading (keep the Pods).</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="kubectl delete petset web --cascade=false"><pre class="notranslate">kubectl delete petset web --cascade=false</pre></div>
</li>
<li>
<p dir="auto">Upgrade master and nodes to <code class="notranslate">v1.5.0-beta.2</code> with <code class="notranslate">cluster/gce/upgrade.sh</code>.</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="git checkout release-1.5
git pull upstream release-1.5
cluster/gce/upgrade.sh v1.5.0-beta.2"><pre class="notranslate">git checkout release-1.5
git pull upstream release-1.5
cluster/gce/upgrade.sh v1.5.0-beta.2</pre></div>
</li>
<li>
<p dir="auto">Check for Pods stuck in ContainerCreating.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME      READY     STATUS              RESTARTS   AGE
web-0     0/1       ContainerCreating   0          42m
web-4     0/1       ContainerCreating   0          42m"><pre class="notranslate"><code class="notranslate">NAME      READY     STATUS              RESTARTS   AGE
web-0     0/1       ContainerCreating   0          42m
web-4     0/1       ContainerCreating   0          42m
</code></pre></div>
</li>
</ol>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">After the above steps, some Pods may end up terminating. It seems to be random which ones terminate and which ones get stuck in ContainerCreating.</p>
<p dir="auto">Manually deleting and recreating all the stuck Pods after the Nodes are upgraded allows them to correctly attach to the preexisting PVCs and become Running.</p>