events.action: opened
events.author: tosi3k
events.text: **Which jobs are failing**:
  `ci-npd-e2e-node`
  
  **Which test(s) are failing**:
  `SystemLogMonitor should generate node condition and events for corresponding errors`
  
  **Since when has it been failing**:
  Last successful run (23 Oct 2020, 10:07 PDT): https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-npd-e2e-node/1319686617375444992
  First unsuccessful run (23 Oct 2020, 12:08 PDT): https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-npd-e2e-node/1319717073798041600
  
  **Testgrid link**:
  https://testgrid.k8s.io/sig-node-node-problem-detector#ci-npd-e2e-node
  
  **Reason for failure**:
  ```
  _output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_problem_detector_linux.go:245
  Timed out after 60.000s.
  Expected success, but got an error:
      <*errors.errorString | 0xc001212cd0>: {
          s: "node condition \"TestCondition\" not found",
      }
      node condition "TestCondition" not found
  _output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_problem_detector_linux.go:360
  ```
  
  **Anything else we need to know**:
  [pull-npd-e2e-node](https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-npd-e2e-node) presubmit job in k8s.io/node-problem-detector repository also started to fail in the same fashion.
  
  Attempted to debug it myself, found out the following stuff:
  * last successful run was accessing the apiserver via HTTP:
  ```
  W1023 17:12:40.514888    2808 test_context.go:420] Unable to find in-cluster config, using default host : http://127.0.0.1:8080
  ```
  * all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail:
  ```
  W1023 19:14:09.903857     995 test_context.go:437] Unable to find in-cluster config, using default host : https://127.0.0.1:6443
  (...)
  W1023 19:15:40.789213     925 util.go:106] Health check on "https://127.0.0.1:6443/healthz" failed, status=500
  ```
  * perused the history of `k/k/test` to find out that the PR https://github.com/kubernetes/kubernetes/pull/94723 is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests):
    * first, it coincides with the start of the failures of `ci-npd-e2e-node`,
    * second, we changed the default hostname of the apiserver in node tests in that PR.
  
  Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a `--host=http://127.0.0.1:8080` flag to the `--node-test-args` of these jobs.
  
  /assign
  /sig node
  /cc @knight42 @Random-Liu
events.title: [Failing test] SystemLogMonitor should generate node condition and events for corresponding errors
events.type: issue
events.action: created
events.author: tosi3k
events.text: Opened https://github.com/kubernetes/test-infra/pull/19735 for a mitigation that would make those jobs stop failing.
events.type: comment
events.action: created
events.author: tosi3k
events.type: comment
events.action: created
events.author: knight42
events.text: @tosi3k I have filed #96050 to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
events.type: comment
events.action: created
events.author: tosi3k
events.text: Unfortunately I don't :/. My PR was also a wild guess. @Random-Liu could you take a look at the proposed fix?
  
  Thanks once again for your help - really appreciated :).
events.type: comment
events.action: created
events.author: tosi3k
events.text: @knight42 @Random-Liu the submission of #96050 hasn't fixed the problems, unfortunately :/.
  
  Sample failing run: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-npd-e2e-node/1324222964127764480
  
  We still have some peculiar problem related to health checking:
  ```
  W1105 05:45:46.863813     983 util.go:106] Health check on "https://127.0.0.1:6443/healthz" failed, status=500
  ```
  
  and now NPD panics, too:
  ```
  panic: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
  goroutine 1 [running]:
  k8s.io/node-problem-detector/pkg/exporters/k8sexporter/problemclient.NewClientOrDie(0xc0003eb790, 0xc0000b6d00, 0x0)
  	/home/prow/go/src/k8s.io/node-problem-detector/pkg/exporters/k8sexporter/problemclient/problem_client.go:73 +0x3f7
  k8s.io/node-problem-detector/pkg/exporters/k8sexporter.NewExporterOrDie(0xc0003eb790, 0xc000492370, 0x1)
  	/home/prow/go/src/k8s.io/node-problem-detector/pkg/exporters/k8sexporter/k8s_exporter.go:52 +0x4f
  main.main()
  	/home/prow/go/src/k8s.io/node-problem-detector/cmd/nodeproblemdetector/node_problem_detector.go:60 +0x13a
  ```
events.type: comment
events.action: created
events.author: knight42
events.text: @tosi3k Thanks for your report, I believe the reason is `ServiceAccount ` admission controller has been disabled in node e2e tests:
  https://github.com/kubernetes/kubernetes/blob/4364ef62062d3d18b0a0cec117af9620f8a896ff/test/e2e_node/services/apiserver.go#L64
  We need to figure out a new approach now.
  
  /reopen
events.type: comment
events.action: created
events.author: knight42
events.text: filed #96262. I hope we could completely address this issue this time.
events.type: comment
events.action: created
events.author: dims
events.text: /milestone v1.20
events.type: comment
data_with_bots.meta: This message directly preceded comment from bot *k8s-ci-robot* by author *dims*:
data_with_bots.text: /milestone v1.20
data_without_bots.meta: This message directly preceded comment from bot *k8s-ci-robot* by author *dims*:
data_without_bots.text: /milestone v1.20