<p dir="auto">FYI. we should add a secret manager to store the pull secrets for different namespaces. Thus for duplicate secrets, we have no need to talk to api server. Actually I am working on this manager, but I have no idea about when to remove a secret.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
v1.4.0</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: GCE + custome kubernetes</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): "CoreOS 1068.8.0 (MoreOS)"</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.6.3-coreos</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
kube-scheduler entered keeps entering restarting due to nil pointer exception. It seems like it happens when a node or pod gets removed from the cluster.<br>
Here is the journal output for the kube-scheduler</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="event.go:217] Event(api.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;default&quot;, Name:&quot;service1-4068773324-24s7s&quot;, UID:&quot;1268223f-864d-11e6-80e7-42010a140003&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;47435&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Scheduled' Successfully assigned service1-4068773324-24s7s to h-stg-wk1-3jvo
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
event.go:217] Event(api.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;default&quot;, Name:&quot;service2-3685785901-549bk&quot;, UID:&quot;12b068d6-864d-11e6-80e7-42010a140003&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;47449&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Scheduled' Successfully assigned service2-3685785901-549bk to h-stg-wk1-vkym
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
event.go:217] Event(api.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;default&quot;, Name:&quot;service3-1539738932-4zdq2&quot;, UID:&quot;8ffd5a12-864d-11e6-80e7-42010a140003&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;50531&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Scheduled' Successfully assigned service3-1539738932-4zdq2 to h-stg-wk1-y8gt
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
runtime.go:64] Observed a panic: &quot;invalid memory address or nil pointer dereference&quot; (runtime error: invalid memory address or nil pointer dereference)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/asm_amd64.s:479
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:458
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:62
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/sigpanic_unix.go:24
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:192
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:229
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:252
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:196
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:134
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:182
Sep 29 14:04:15 node-123 docker[15022]: &lt;autogenerated&gt;:52
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:318
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:126
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/asm_amd64.s:2086
Sep 29 14:04:15 node-123 docker[15022]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Sep 29 14:04:15 node-123 docker[15022]:         panic: runtime error: invalid memory address or nil pointer dereference
Sep 29 14:04:15 node-123 docker[15022]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1aa376c]
Sep 29 14:04:15 node-123 docker[15022]: goroutine 16 [running]:
Sep 29 14:04:15 node-123 docker[15022]: panic(0x2ccdf20, 0xc420016050)
Sep 29 14:04:15 node-123 docker[15022]:         /usr/lib/go/src/runtime/panic.go:500 +0x1a1
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126
Sep 29 14:04:15 node-123 docker[15022]: panic(0x2ccdf20, 0xc420016050)
Sep 29 14:04:15 node-123 docker[15022]:         /usr/lib/go/src/runtime/panic.go:458 +0x243
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*NodeInfo).removePod(0x0, 0xc420eacc80, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:192 +0x7c
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*schedulerCache).removePod(0xc420ce3d40, 0xc420eacc80, 0xc4210509a0, 0x1d)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:229 +0x88
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*schedulerCache).RemovePod(0xc420ce3d40, 0xc420eacc80, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:252 +0x22d
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).deletePodFromCache(0xc42009ebb0, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:196 +0xa9
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).(k8s.io/kubernetes/plugin/pkg/scheduler/factory.deletePodFromCache)-fm(0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:134 +0x3e
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.ResourceEventHandlerFuncs.OnDelete(0xc4203f31d0, 0xc4203f31f0, 0xc4203f3200, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:182 +0x49
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc420d5cb20, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         &lt;autogenerated&gt;:52 +0x78
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.NewIndexerInformer.func1(0x2d35cc0, 0xc421050960, 0xc421050960, 0x2d35cc0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:318 +0x4ee
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).Pop(0xc42009ec60, 0xc4200152c0, 0x0, 0x0, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420 +0x22a
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).processLoop(0xc420150150)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:126 +0x3c
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).(k8s.io/kubernetes/pkg/client/cache.processLoop)-fm()
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x2a
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc4210d7f70)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84 +0x19
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc4210d7f70, 0x3b9aca00, 0x0, 0x2b48701, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85 +0xad
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.Until(0xc4210d7f70, 0x3b9aca00, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x4d
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).Run(0xc420150150, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x1af
Sep 29 14:04:15 node-123 docker[15022]: created by k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).Run
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:391 +0x12f
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Unit entered failed state.
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Failed with result 'exit-code'."><pre class="notranslate"><code class="notranslate">event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"service1-4068773324-24s7s", UID:"1268223f-864d-11e6-80e7-42010a140003", APIVersion:"v1", ResourceVersion:"47435", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned service1-4068773324-24s7s to h-stg-wk1-3jvo
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"service2-3685785901-549bk", UID:"12b068d6-864d-11e6-80e7-42010a140003", APIVersion:"v1", ResourceVersion:"47449", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned service2-3685785901-549bk to h-stg-wk1-vkym
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"service3-1539738932-4zdq2", UID:"8ffd5a12-864d-11e6-80e7-42010a140003", APIVersion:"v1", ResourceVersion:"50531", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned service3-1539738932-4zdq2 to h-stg-wk1-y8gt
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
listers.go:68] can not retrieve list of objects using index : object has no meta: object does not implement the Object interfaces
runtime.go:64] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:70
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:63
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:49
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/asm_amd64.s:479
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:458
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/panic.go:62
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/sigpanic_unix.go:24
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:192
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:229
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:252
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:196
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:134
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:182
Sep 29 14:04:15 node-123 docker[15022]: &lt;autogenerated&gt;:52
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:318
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:126
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47
Sep 29 14:04:15 node-123 docker[15022]: /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102
Sep 29 14:04:15 node-123 docker[15022]: /usr/lib/go/src/runtime/asm_amd64.s:2086
Sep 29 14:04:15 node-123 docker[15022]: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Sep 29 14:04:15 node-123 docker[15022]:         panic: runtime error: invalid memory address or nil pointer dereference
Sep 29 14:04:15 node-123 docker[15022]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1aa376c]
Sep 29 14:04:15 node-123 docker[15022]: goroutine 16 [running]:
Sep 29 14:04:15 node-123 docker[15022]: panic(0x2ccdf20, 0xc420016050)
Sep 29 14:04:15 node-123 docker[15022]:         /usr/lib/go/src/runtime/panic.go:500 +0x1a1
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/runtime/runtime.go:56 +0x126
Sep 29 14:04:15 node-123 docker[15022]: panic(0x2ccdf20, 0xc420016050)
Sep 29 14:04:15 node-123 docker[15022]:         /usr/lib/go/src/runtime/panic.go:458 +0x243
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*NodeInfo).removePod(0x0, 0xc420eacc80, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/node_info.go:192 +0x7c
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*schedulerCache).removePod(0xc420ce3d40, 0xc420eacc80, 0xc4210509a0, 0x1d)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:229 +0x88
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache.(*schedulerCache).RemovePod(0xc420ce3d40, 0xc420eacc80, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache/cache.go:252 +0x22d
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).deletePodFromCache(0xc42009ebb0, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:196 +0xa9
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).(k8s.io/kubernetes/plugin/pkg/scheduler/factory.deletePodFromCache)-fm(0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:134 +0x3e
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.ResourceEventHandlerFuncs.OnDelete(0xc4203f31d0, 0xc4203f31f0, 0xc4203f3200, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:182 +0x49
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc420d5cb20, 0x31e0340, 0xc420eacc80)
Sep 29 14:04:15 node-123 docker[15022]:         &lt;autogenerated&gt;:52 +0x78
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.NewIndexerInformer.func1(0x2d35cc0, 0xc421050960, 0xc421050960, 0x2d35cc0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:318 +0x4ee
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).Pop(0xc42009ec60, 0xc4200152c0, 0x0, 0x0, 0x0, 0x0)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:420 +0x22a
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).processLoop(0xc420150150)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:126 +0x3c
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).(k8s.io/kubernetes/pkg/client/cache.processLoop)-fm()
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x2a
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc4210d7f70)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:84 +0x19
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc4210d7f70, 0x3b9aca00, 0x0, 0x2b48701, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:85 +0xad
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/util/wait.Until(0xc4210d7f70, 0x3b9aca00, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/util/wait/wait.go:47 +0x4d
Sep 29 14:04:15 node-123 docker[15022]: k8s.io/kubernetes/pkg/client/cache.(*Controller).Run(0xc420150150, 0xc420d58780)
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/pkg/client/cache/controller.go:102 +0x1af
Sep 29 14:04:15 node-123 docker[15022]: created by k8s.io/kubernetes/plugin/pkg/scheduler/factory.(*ConfigFactory).Run
Sep 29 14:04:15 node-123 docker[15022]:         /home/dima/Private/Work/k8s/src/k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:391 +0x12f
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Unit entered failed state.
Sep 29 14:04:15 node-123 systemd[1]: kube-scheduler@node-123.service: Failed with result 'exit-code'.
</code></pre></div>
<p dir="auto">Based on the source code it looks like here:<br>
<a href="https://github.com/kubernetes/kubernetes/blob/release-1.4/plugin/pkg/scheduler/schedulercache/cache.go#L228">https://github.com/kubernetes/kubernetes/blob/release-1.4/plugin/pkg/scheduler/schedulercache/cache.go#L228</a></p>
<p dir="auto">we get pod from cache without checking if it actually exists there. Since it is a pointer we can call <code class="notranslate">removePod</code> on it. But once inside <a href="https://github.com/kubernetes/kubernetes/blob/release-1.4/plugin/pkg/scheduler/schedulercache/node_info.go#L192">https://github.com/kubernetes/kubernetes/blob/release-1.4/plugin/pkg/scheduler/schedulercache/node_info.go#L192</a> we try to get filed of a nil pointer and so it crashes.</p>
<p dir="auto"><strong>What you expected to happen</strong>: Expected not to get a nil pointer exception.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
This is also related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="174743861" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/31968" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/31968/hovercard" href="https://github.com/kubernetes/kubernetes/issues/31968">#31968</a> but this time exception is in a different place. Also it looks like this might cause some of our pods stay in <code class="notranslate">pending</code> state indefinitely but I am not sure.</p>