Answers: <p>The files in <code>/etc/kubernetes/pki</code> are found on the API server, not on the nodes. And as you've noted the <code>--tls-cert-file=/srv/kubernetes/kubelet-server.crt</code> parameter of your Kubelet specifies which certificate file it is using.</p>
  <p>One possible problem you might have is time drift. In order to verify certificates, both server and client need to have synchronized clocks. You can try and sync your node with NTP and perhaps this will resolve the issue.</p>
  <p>Another potential problem is that the Certificate Authority, one of the files in <code>/etc/kubernetes/pki</code> on the API server, is not the same CA that signed your kubelet's certificate. When a Kubelet is created, it receives a certificate signed by this specific Kubernetes API server, and that is how they authenticate with each other. But if for example you try and authenticate this Kubelet with a different service, it will not recognize it. While a Kubelet is running, new refreshed certificates are issued from time to time and applied on the Kubelet side. You can inspect the Kubelet certificate and the CA certificate to make sure they match. You noted that the certificate is not expired, but it might have been revoked, this can also be seen on the side of the API server in its revocation list.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-06-25T10:05:21.670Z
Answers: 2023-06-25T10:05:21.670Z
Answers: Evgeny Zislis
Body: <p>I have a cluster that is setup using the tool <code>kops</code> version 1.21.5. I'm aware that's quite old. The nodes have been up for quite some time now (eg; ~450 days)</p>
  <p>I'm receiving the following alert &quot;A client certificate used to authenticate to kubernetes apiserver is expiring in less than 7.0 days&quot; which I believe ships by default with prometheus operator.</p>
  <p>I've had a look on the nodes and they are running <code>kubelet</code> with the following flags</p>
  <pre><code>/usr/local/bin/kubelet --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook ... --kubeconfig=/var/lib/kubelet/kubeconfig ...--tls-cert-file=/srv/kubernetes/kubelet-server.crt --tls-private-key-file=/srv/kubernetes/kubelet-server.key
  </code></pre>
  <p>I believe this means they should be attempting to authenticate with the cert/key from either those files or the one stored in the <code>kubeconfig</code> (a bit unsure on this point).</p>
  <p>However when I check those certificates using <code>openssl x509 --noout -text</code> they all seem to have sensible values in the <code>notAfter</code></p>
  <p>I've had a node go down, whereby it'll start complaining about 'Unauthorized' when trying to access the kube-apiserver. When this happened, I was able to confirm that there is definitely an x509 certificate being sent to the api-server that is past its 'notAfter' date, based on the logs from the kube-apiserver pod. The 'notAfter' date in the logs is not the date on the .crt/stored cert in the kubeconfig.</p>
  <p>On the node in /var/log/syslog;</p>
  <pre><code>&quot;Error updating node status, will retry&quot; err=&quot;error getting node \&quot;&lt;my node name&gt;&quot;: Unauthorized&quot;
  </code></pre>
  <p>In the kubectl logs from the relevant <code>kube-apiserver</code></p>
  <pre><code>&quot;Unable to authenticate the request&quot; err=&quot;[x509: certificate has expired or is not yet valid: current time 2023-06-18T02:18:16Z is after 2023-06-16T14:46:43Z, verifying certificate 
  </code></pre>
  <p>I've tried to <code>find</code> everything ending in .crt and didn't see anything obvious. There were the two from above, a bunch of <code>mozilla</code> ones which at a guess come from Ubuntu by default and a lot in /docker/overlay2 which I guess are something from the containers.</p>
  <p>Terminating the nodes in AWS seems to result in a new node being created which has a valid expiry that doesn't trip the warning anymore. So something must be getting recreated somewhere.</p>
  <p>I'd really like to know where this cert is, but everything I search for online has pointed me to either those files or <code>/etc/kubernetes/pki</code> (which doesn't seem to exist on my workers)</p>
  
Tags: <kubernetes><kops>
Title: Where should I look for the x509 certificate Kubelet is using to communicate with the kube-apiserver?
UserDisplayName: jwfischer
ThreadText: # Where should I look for the x509 certificate Kubelet is using to communicate with the kube-apiserver?
  
  (Asked by: username_0 on 2023-06-20)
  
  I have a cluster that is setup using the tool `kops` version 1.21.5. I'm aware that's quite old. The nodes have been up for quite some time now (eg; ~450 days)
  
  I'm receiving the following alert "A client certificate used to authenticate to kubernetes apiserver is expiring in less than 7.0 days" which I believe ships by default with prometheus operator.
  
  I've had a look on the nodes and they are running `kubelet` with the following flags
  
  ```
  /usr/local/bin/kubelet --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook ... --kubeconfig=/var/lib/kubelet/kubeconfig ...--tls-cert-file=/srv/kubernetes/kubelet-server.crt --tls-private-key-file=/srv/kubernetes/kubelet-server.key
  
  ```
  
  I believe this means they should be attempting to authenticate with the cert/key from either those files or the one stored in the `kubeconfig` (a bit unsure on this point).
  
  However when I check those certificates using `openssl x509 --noout -text` they all seem to have sensible values in the `notAfter`
  
  I've had a node go down, whereby it'll start complaining about 'Unauthorized' when trying to access the kube-apiserver. When this happened, I was able to confirm that there is definitely an x509 certificate being sent to the api-server that is past its 'notAfter' date, based on the logs from the kube-apiserver pod. The 'notAfter' date in the logs is not the date on the .crt/stored cert in the kubeconfig.
  
  On the node in /var/log/syslog;
  
  ```
  "Error updating node status, will retry" err="error getting node \"<my node name>": Unauthorized"
  
  ```
  
  In the kubectl logs from the relevant `kube-apiserver`
  
  ```
  "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2023-06-18T02:18:16Z is after 2023-06-16T14:46:43Z, verifying certificate 
  
  ```
  
  I've tried to `find` everything ending in .crt and didn't see anything obvious. There were the two from above, a bunch of `mozilla` ones which at a guess come from Ubuntu by default and a lot in /docker/overlay2 which I guess are something from the containers.
  
  Terminating the nodes in AWS seems to result in a new node being created which has a valid expiry that doesn't trip the warning anymore. So something must be getting recreated somewhere.
  
  I'd really like to know where this cert is, but everything I search for online has pointed me to either those files or `/etc/kubernetes/pki` (which doesn't seem to exist on my workers)
  
  **Tags:** kubernetes, kops
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-25. Score: 0
  
  The files in `/etc/kubernetes/pki` are found on the API server, not on the nodes. And as you've noted the `--tls-cert-file=/srv/kubernetes/kubelet-server.crt` parameter of your Kubelet specifies which certificate file it is using.
  
  One possible problem you might have is time drift. In order to verify certificates, both server and client need to have synchronized clocks. You can try and sync your node with NTP and perhaps this will resolve the issue.
  
  Another potential problem is that the Certificate Authority, one of the files in `/etc/kubernetes/pki` on the API server, is not the same CA that signed your kubelet's certificate. When a Kubelet is created, it receives a certificate signed by this specific Kubernetes API server, and that is how they authenticate with each other. But if for example you try and authenticate this Kubelet with a different service, it will not recognize it. While a Kubelet is running, new refreshed certificates are issued from time to time and applied on the Kubelet side. You can inspect the Kubelet certificate and the CA certificate to make sure they match. You noted that the certificate is not expired, but it might have been revoked, this can also be seen on the side of the API server in its revocation list.
  
  
  ---
  
  