<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">Yes</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">runtime.go:64] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)</p>
<p dir="auto">kubernetes arm invalid memory etcdWatcher</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">1.4</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Raspbery Pi 3</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.1.19-v7 armv7l</li>
<li><strong>Install tools</strong>: kubeadm</li>
<li><strong>Others</strong>:  Had to add /boot/cmdline.txt added cgroup_enable=memory</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">When kubeadm starts the api server it starts, begins listening then fails with the following message:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0929 15:21:31.213981       1 genericapiserver.go:717] Serving securely on 0.0.0.0:443
I0929 15:21:31.214120       1 genericapiserver.go:762] Serving insecurely on 127.0.0.1:8080
I0929 15:21:31.227785       1 reflector.go:249] Listing and watching *api.LimitRange from k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154
I0929 15:21:31.233122       1 handlers.go:162] GET /api/v1/limitranges?resourceVersion=0: (2.872585ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47712]
I0929 15:21:31.233647       1 reflector.go:249] Listing and watching *api.ServiceAccount from k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103
I0929 15:21:31.234397       1 reflector.go:249] Listing and watching *storage.StorageClass from k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62
I0929 15:21:31.234689       1 reflector.go:249] Listing and watching *api.ResourceQuota from k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83
I0929 15:21:31.235308       1 reflector.go:249] Listing and watching *api.Secret from k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119
I0929 15:21:31.240989       1 handlers.go:162] GET /api/v1/resourcequotas?resourceVersion=0: (2.16535ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47712]
I0929 15:21:31.241224       1 handlers.go:162] GET /api/v1/serviceaccounts?resourceVersion=0: (3.223572ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47713]
I0929 15:21:31.242460       1 handlers.go:162] GET /apis/storage.k8s.io/v1beta1/storageclasses?resourceVersion=0: (3.584299ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47714]
I0929 15:21:31.243120       1 handlers.go:162] GET /api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&amp;resourceVersion=0: (2.397484ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47717]
I0929 15:21:31.385736       1 handlers.go:162] GET /api/v1/nodes?resourceVersion=0: (2.577587ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47720]
I0929 15:21:31.398981       1 handlers.go:162] GET /api/v1/services?resourceVersion=0: (1.901341ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47721]
I0929 15:21:31.429622       1 logs.go:41] http: TLS handshake error from 192.168.3.50:44034: read tcp 192.168.3.50:443-&gt;192.168.3.50:44034: read: connection reset by peer
I0929 15:21:31.526282       1 handlers.go:162] GET /api/v1/replicationcontrollers?resourceVersion=0: (4.753822ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47723]
I0929 15:21:31.596865       1 handlers.go:162] GET /api/v1/persistentvolumes?resourceVersion=0: (5.896991ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47724]
I0929 15:21:31.598638       1 handlers.go:162] GET /api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: (7.20141ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47727]
I0929 15:21:31.605112       1 handlers.go:162] GET /api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: (6.404489ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47726]
I0929 15:21:31.646091       1 handlers.go:162] GET /apis/extensions/v1beta1/replicasets?resourceVersion=0: (43.100753ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47725]
I0929 15:21:31.692673       1 reflector.go:249] Listing and watching *api.Namespace from ????
I0929 15:21:31.703559       1 handlers.go:162] GET /api/v1/namespaces?resourceVersion=0: (6.497352ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47716]
I0929 15:21:31.743619       1 logs.go:41] http: TLS handshake error from 192.168.3.50:44037: read tcp 192.168.3.50:443-&gt;192.168.3.50:44037: read: connection reset by peer
I0929 15:21:31.822912       1 logs.go:41] http: TLS handshake error from 192.168.3.50:44043: read tcp 192.168.3.50:443-&gt;192.168.3.50:44043: read: connection reset by peer
I0929 15:21:32.223161       1 handlers.go:162] GET /api/v1/persistentvolumeclaims?resourceVersion=0: (4.613355ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47729]
I0929 15:21:32.355696       1 handlers.go:162] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (6.227823ms) 200 [[kube-controller-manager/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47730]
I0929 15:21:32.380576       1 handlers.go:162] PUT /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (16.689993ms) 200 [[kube-controller-manager/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47730]
E0929 15:21:32.380501       1 runtime.go:64] Observed a panic: &quot;invalid memory address or nil pointer dereference&quot; (runtime error: invalid memory address or nil pointer dereference)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
/usr/local/go_k8s_patched/src/runtime/asm_arm.s:429
/usr/local/go_k8s_patched/src/runtime/panic.go:458
/usr/local/go_k8s_patched/src/runtime/panic.go:62
/usr/local/go_k8s_patched/src/runtime/sigpanic_unix.go:24
/usr/local/go_k8s_patched/src/sync/atomic/asm_linux_arm.s:109
/usr/local/go_k8s_patched/src/sync/atomic/64bit_arm.go:10
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:57
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:316
/usr/local/go_k8s_patched/src/runtime/asm_arm.s:998
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11fa64]

goroutine 212 [running]:
panic(0x2799878, 0x14022008)
    /usr/local/go_k8s_patched/src/runtime/panic.go:500 +0x3f4
k8s.io/kubernetes/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x170
panic(0x2799878, 0x14022008)
    /usr/local/go_k8s_patched/src/runtime/panic.go:458 +0x51c
sync/atomic.loadUint64(0x1435d764, 0x0, 0x0)
    /usr/local/go_k8s_patched/src/sync/atomic/64bit_arm.go:10 +0x58
k8s.io/kubernetes/pkg/storage/etcd.(*HighWaterMark).Update(0x1435d764, 0x0, 0x0, 0x147066a8)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:57 +0x24
k8s.io/kubernetes/pkg/storage/etcd.(*etcdWatcher).translate(0x1435d700)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:316 +0x564
created by k8s.io/kubernetes/pkg/storage/etcd.newEtcdWatcher
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:172 +0x2fc"><pre class="notranslate"><code class="notranslate">I0929 15:21:31.213981       1 genericapiserver.go:717] Serving securely on 0.0.0.0:443
I0929 15:21:31.214120       1 genericapiserver.go:762] Serving insecurely on 127.0.0.1:8080
I0929 15:21:31.227785       1 reflector.go:249] Listing and watching *api.LimitRange from k8s.io/kubernetes/plugin/pkg/admission/limitranger/admission.go:154
I0929 15:21:31.233122       1 handlers.go:162] GET /api/v1/limitranges?resourceVersion=0: (2.872585ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47712]
I0929 15:21:31.233647       1 reflector.go:249] Listing and watching *api.ServiceAccount from k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:103
I0929 15:21:31.234397       1 reflector.go:249] Listing and watching *storage.StorageClass from k8s.io/kubernetes/plugin/pkg/admission/storageclass/default/admission.go:62
I0929 15:21:31.234689       1 reflector.go:249] Listing and watching *api.ResourceQuota from k8s.io/kubernetes/plugin/pkg/admission/resourcequota/resource_access.go:83
I0929 15:21:31.235308       1 reflector.go:249] Listing and watching *api.Secret from k8s.io/kubernetes/plugin/pkg/admission/serviceaccount/admission.go:119
I0929 15:21:31.240989       1 handlers.go:162] GET /api/v1/resourcequotas?resourceVersion=0: (2.16535ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47712]
I0929 15:21:31.241224       1 handlers.go:162] GET /api/v1/serviceaccounts?resourceVersion=0: (3.223572ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47713]
I0929 15:21:31.242460       1 handlers.go:162] GET /apis/storage.k8s.io/v1beta1/storageclasses?resourceVersion=0: (3.584299ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47714]
I0929 15:21:31.243120       1 handlers.go:162] GET /api/v1/secrets?fieldSelector=type%3Dkubernetes.io%2Fservice-account-token&amp;resourceVersion=0: (2.397484ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47717]
I0929 15:21:31.385736       1 handlers.go:162] GET /api/v1/nodes?resourceVersion=0: (2.577587ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47720]
I0929 15:21:31.398981       1 handlers.go:162] GET /api/v1/services?resourceVersion=0: (1.901341ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47721]
I0929 15:21:31.429622       1 logs.go:41] http: TLS handshake error from 192.168.3.50:44034: read tcp 192.168.3.50:443-&gt;192.168.3.50:44034: read: connection reset by peer
I0929 15:21:31.526282       1 handlers.go:162] GET /api/v1/replicationcontrollers?resourceVersion=0: (4.753822ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47723]
I0929 15:21:31.596865       1 handlers.go:162] GET /api/v1/persistentvolumes?resourceVersion=0: (5.896991ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47724]
I0929 15:21:31.598638       1 handlers.go:162] GET /api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: (7.20141ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47727]
I0929 15:21:31.605112       1 handlers.go:162] GET /api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&amp;resourceVersion=0: (6.404489ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47726]
I0929 15:21:31.646091       1 handlers.go:162] GET /apis/extensions/v1beta1/replicasets?resourceVersion=0: (43.100753ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47725]
I0929 15:21:31.692673       1 reflector.go:249] Listing and watching *api.Namespace from ????
I0929 15:21:31.703559       1 handlers.go:162] GET /api/v1/namespaces?resourceVersion=0: (6.497352ms) 200 [[kube-apiserver/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47716]
I0929 15:21:31.743619       1 logs.go:41] http: TLS handshake error from 192.168.3.50:44037: read tcp 192.168.3.50:443-&gt;192.168.3.50:44037: read: connection reset by peer
I0929 15:21:31.822912       1 logs.go:41] http: TLS handshake error from 192.168.3.50:44043: read tcp 192.168.3.50:443-&gt;192.168.3.50:44043: read: connection reset by peer
I0929 15:21:32.223161       1 handlers.go:162] GET /api/v1/persistentvolumeclaims?resourceVersion=0: (4.613355ms) 200 [[kube-scheduler/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47729]
I0929 15:21:32.355696       1 handlers.go:162] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (6.227823ms) 200 [[kube-controller-manager/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47730]
I0929 15:21:32.380576       1 handlers.go:162] PUT /api/v1/namespaces/kube-system/endpoints/kube-controller-manager: (16.689993ms) 200 [[kube-controller-manager/v1.4.0 (linux/arm) kubernetes/a16c0a7] 127.0.0.1:47730]
E0929 15:21:32.380501       1 runtime.go:64] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
/usr/local/go_k8s_patched/src/runtime/asm_arm.s:429
/usr/local/go_k8s_patched/src/runtime/panic.go:458
/usr/local/go_k8s_patched/src/runtime/panic.go:62
/usr/local/go_k8s_patched/src/runtime/sigpanic_unix.go:24
/usr/local/go_k8s_patched/src/sync/atomic/asm_linux_arm.s:109
/usr/local/go_k8s_patched/src/sync/atomic/64bit_arm.go:10
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:57
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:316
/usr/local/go_k8s_patched/src/runtime/asm_arm.s:998
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11fa64]

goroutine 212 [running]:
panic(0x2799878, 0x14022008)
    /usr/local/go_k8s_patched/src/runtime/panic.go:500 +0x3f4
k8s.io/kubernetes/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x170
panic(0x2799878, 0x14022008)
    /usr/local/go_k8s_patched/src/runtime/panic.go:458 +0x51c
sync/atomic.loadUint64(0x1435d764, 0x0, 0x0)
    /usr/local/go_k8s_patched/src/sync/atomic/64bit_arm.go:10 +0x58
k8s.io/kubernetes/pkg/storage/etcd.(*HighWaterMark).Update(0x1435d764, 0x0, 0x0, 0x147066a8)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:57 +0x24
k8s.io/kubernetes/pkg/storage/etcd.(*etcdWatcher).translate(0x1435d700)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:316 +0x564
created by k8s.io/kubernetes/pkg/storage/etcd.newEtcdWatcher
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd/etcd_watcher.go:172 +0x2fc
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">kubeadm finished installing</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ol dir="auto">
<li>Deploy ubuntu mate on raspberry pi 3</li>
<li>apt-get update, apt-get upgrade</li>
<li>add cgroup_enable=memory to /boot/cmdline.txt</li>
<li>install packages per <a href="http://kubernetes.io/docs/getting-started-guides/kubeadm/" rel="nofollow">http://kubernetes.io/docs/getting-started-guides/kubeadm/</a></li>
</ol>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">Kubernetes scheduler schedules based on resources managed by Kubernetes. Scheduling based on opaque resource counting helps extending this further. But when there is a need for contextual scheduling for resources managed outside of kubernetes(example: place a pod where its storage is), there is no mechanism to do it today.</p>
<p dir="auto">The proposal is to make kubernetes scheduler extensible by adding the capability to make http calls out to another endpoint to help achieve this functionality. I am curious whether you think the cloud provider abstraction is the right abstraction for implementation.</p>
<p dir="auto">Here is a rough draft of what I am thinking about. Would like to solicit community feedback</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="type SchedulerExtension interface {

      // Filter based on provider implemented predicate functions.
      Filter(pod *api.Pod, nodes *api.NodeList) (*api.NodeList, error)

      // Prioritize based on provider implemented priority functions. Weight*priority is added up for each
      // such priority function. The returned score is added to the score computed by Kubernetes
      // scheduler. The total score is used to do the host selection.
      Prioritize(pod *api.Pod, nodes *api.NodeList) (*scheduler/api.HostPriorityList, error)

      // Inform the provider about the scheduling decision. This could also be done by the provider
      // watching apiserver pods/binding endpoint.
      Bind(pod *api.Pod, host string) error

      // Inform the provider about the unbind. To be called by apiserver in pod deletion path. This could
      // also be accomplished by watching apiserver pods endpoint.
      Unbind(pod *api.Pod, host string) error
}"><pre class="notranslate"><code class="notranslate">type SchedulerExtension interface {

      // Filter based on provider implemented predicate functions.
      Filter(pod *api.Pod, nodes *api.NodeList) (*api.NodeList, error)

      // Prioritize based on provider implemented priority functions. Weight*priority is added up for each
      // such priority function. The returned score is added to the score computed by Kubernetes
      // scheduler. The total score is used to do the host selection.
      Prioritize(pod *api.Pod, nodes *api.NodeList) (*scheduler/api.HostPriorityList, error)

      // Inform the provider about the scheduling decision. This could also be done by the provider
      // watching apiserver pods/binding endpoint.
      Bind(pod *api.Pod, host string) error

      // Inform the provider about the unbind. To be called by apiserver in pod deletion path. This could
      // also be accomplished by watching apiserver pods endpoint.
      Unbind(pod *api.Pod, host string) error
}
</code></pre></div>