Answers: <p>You need to send bearer token as well, like this:</p>
  <pre><code>- job_name: 'kubelet'
          kubernetes_sd_configs:
          - role: node
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: true 
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2023-03-13T01:16:11.340Z
Answers: Thanks Shanmuga! That changed the response from "server returned HTTP status 401 Unauthorized" to "server returned HTTP status 403 Forbidden". 
  
  However, I still don't know why it is not authorized
Answers: codedread
Answers: CC BY-SA 4.0
Answers: 2023-03-05T18:32:21.733Z
Answers: 2023-03-05T18:32:48.880Z
Answers: 2023-03-05T18:32:48.880Z
Answers: Shanmuga Raj
Body: <p>I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this:</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Namespace
  metadata:
      name: monitoring
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: prometheus
  rules:
  - apiGroups: [&quot;&quot;]
    resources:
    - nodes
    - services
    - endpoints
    - pods
    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
  - apiGroups:
    - extensions
    resources:
    - ingresses
    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
  - nonResourceURLs:
    - &quot;/metrics&quot;
    verbs: [&quot;get&quot;]
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: prometheus
    namespace: monitoring
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: prometheus
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: prometheus
  subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring
  </code></pre>
  <p>with my Deployment resource containing <code>serviceAccountName: prometheus</code>.</p>
  <p>I am getting some metrics (from Prometheus and some Istio scrape configs), but am not able to get the k8s node metrics (like CPU, etc). I tried this:</p>
  <pre class="lang-yaml prettyprint-override"><code>      - job_name: 'kubelet'
          kubernetes_sd_configs:
          - role: node
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  </code></pre>
  <p>I get an error in my Prometheus targets dashboard (<a href="http://192.168.0.121:30000/targets?search=" rel="nofollow noreferrer">http://192.168.0.121:30000/targets?search=</a> of:</p>
  <p><code>Get &quot;https://192.168.0.121:10250/metrics&quot;: x509: cannot validate certificate for 192.168.0.121 because it doesn't contain any IP SANs</code></p>
  <p>Adding <code>insecure_skip_verify: true</code> turns this error into:</p>
  <p><code>server returned HTTP status 401 Unauthorized</code></p>
  <p>My KubeletConfiguration:</p>
  <pre class="lang-bash prettyprint-override"><code>% kubectl describe configmap kubelet-config -n kube-system
  Name:         kubelet-config
  Namespace:    kube-system
  Labels:       &lt;none&gt;
  Annotations:  kubeadm.kubernetes.io/component-config.hash: ...
  
  Data
  ====
  kubelet:
  ----
  apiVersion: kubelet.config.k8s.io/v1beta1
  authentication:
    anonymous:
      enabled: false
    webhook:
      cacheTTL: 0s
      enabled: true
    x509:
      clientCAFile: /etc/kubernetes/pki/ca.crt
  authorization:
    mode: Webhook
    webhook:
      cacheAuthorizedTTL: 0s
      cacheUnauthorizedTTL: 0s
  cgroupDriver: systemd
  clusterDNS:
  - 10.96.0.10
  clusterDomain: cluster.local
  cpuManagerReconcilePeriod: 0s
  evictionPressureTransitionPeriod: 0s
  fileCheckFrequency: 0s
  healthzBindAddress: 127.0.0.1
  healthzPort: 10248
  httpCheckFrequency: 0s
  imageMinimumGCAge: 0s
  kind: KubeletConfiguration
  logging:
    flushFrequency: 0
    options:
      json:
        infoBufferSize: &quot;0&quot;
    verbosity: 0
  memorySwap: {}
  nodeStatusReportFrequency: 0s
  nodeStatusUpdateFrequency: 0s
  resolvConf: /run/systemd/resolve/resolv.conf
  rotateCertificates: true
  runtimeRequestTimeout: 0s
  shutdownGracePeriod: 0s
  shutdownGracePeriodCriticalPods: 0s
  staticPodPath: /etc/kubernetes/manifests
  streamingConnectionIdleTimeout: 0s
  syncFrequency: 0s
  volumeStatsAggPeriod: 0s
  </code></pre>
  <p>My kubeadm-config is:</p>
  <pre class="lang-bash prettyprint-override"><code>% kubectl describe configmap kubeadm-config -n kube-system
  Name:         kubeadm-config
  Namespace:    kube-system
  Labels:       &lt;none&gt;
  Annotations:  &lt;none&gt;
  
  Data
  ====
  ClusterConfiguration:
  ----
  apiServer:
    certSANs:
    - &lt;my tailscale ip address&gt;
    extraArgs:
      authorization-mode: Node,RBAC
    timeoutForControlPlane: 4m0s
  apiVersion: kubeadm.k8s.io/v1beta3
  certificatesDir: /etc/kubernetes/pki
  clusterName: kubernetes
  controlPlaneEndpoint: 192.168.0.121:6443
  controllerManager: {}
  dns: {}
  etcd:
    local:
      dataDir: /var/lib/etcd
  imageRepository: registry.k8s.io
  kind: ClusterConfiguration
  kubernetesVersion: v1.25.3
  networking:
    dnsDomain: cluster.local
    podSubnet: 10.244.0.0/16
    serviceSubnet: 10.96.0.0/12
  scheduler: {}
  </code></pre>
  <p>which was created by</p>
  <p><code>kubeadm init --control-plane-endpoint=192.168.0.121 --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=&lt;my tailnet ip address&gt;</code></p>
  <p>Any insight would be greatly appreciated!</p>
  
Tags: <kubernetes><prometheus><kubelet>
Title: Prometheus scrape config for kubelet node role not auth'ing
UserDisplayName: codedread
ThreadText: # Prometheus scrape config for kubelet node role not auth'ing
  
  (Asked by: username_0 on 2023-01-29)
  
  I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this:
  
  ```yaml
  apiVersion: v1
  kind: Namespace
  metadata:
      name: monitoring
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: prometheus
  rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - endpoints
    - pods
    verbs: ["get", "list", "watch"]
  - apiGroups:
    - extensions
    resources:
    - ingresses
    verbs: ["get", "list", "watch"]
  - nonResourceURLs:
    - "/metrics"
    verbs: ["get"]
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: prometheus
    namespace: monitoring
  ---
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRoleBinding
  metadata:
    name: prometheus
  roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: prometheus
  subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring
  
  ```
  
  with my Deployment resource containing `serviceAccountName: prometheus`.
  
  I am getting some metrics (from Prometheus and some Istio scrape configs), but am not able to get the k8s node metrics (like CPU, etc). I tried this:
  
  ```yaml
        - job_name: 'kubelet'
          kubernetes_sd_configs:
          - role: node
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  
  ```
  
  I get an error in my Prometheus targets dashboard (<http://192.168.0.121:30000/targets?search=> of:
  
  `Get "https://192.168.0.121:10250/metrics": x509: cannot validate certificate for 192.168.0.121 because it doesn't contain any IP SANs`
  
  Adding `insecure_skip_verify: true` turns this error into:
  
  `server returned HTTP status 401 Unauthorized`
  
  My KubeletConfiguration:
  
  ```bash
  % kubectl describe configmap kubelet-config -n kube-system
  Name:         kubelet-config
  Namespace:    kube-system
  Labels:       <none>
  Annotations:  kubeadm.kubernetes.io/component-config.hash: ...
  
  Data
  ====
  kubelet:
  ----
  apiVersion: kubelet.config.k8s.io/v1beta1
  authentication:
    anonymous:
      enabled: false
    webhook:
      cacheTTL: 0s
      enabled: true
    x509:
      clientCAFile: /etc/kubernetes/pki/ca.crt
  authorization:
    mode: Webhook
    webhook:
      cacheAuthorizedTTL: 0s
      cacheUnauthorizedTTL: 0s
  cgroupDriver: systemd
  clusterDNS:
  - 10.96.0.10
  clusterDomain: cluster.local
  cpuManagerReconcilePeriod: 0s
  evictionPressureTransitionPeriod: 0s
  fileCheckFrequency: 0s
  healthzBindAddress: 127.0.0.1
  healthzPort: 10248
  httpCheckFrequency: 0s
  imageMinimumGCAge: 0s
  kind: KubeletConfiguration
  logging:
    flushFrequency: 0
    options:
      json:
        infoBufferSize: "0"
    verbosity: 0
  memorySwap: {}
  nodeStatusReportFrequency: 0s
  nodeStatusUpdateFrequency: 0s
  resolvConf: /run/systemd/resolve/resolv.conf
  rotateCertificates: true
  runtimeRequestTimeout: 0s
  shutdownGracePeriod: 0s
  shutdownGracePeriodCriticalPods: 0s
  staticPodPath: /etc/kubernetes/manifests
  streamingConnectionIdleTimeout: 0s
  syncFrequency: 0s
  volumeStatsAggPeriod: 0s
  
  ```
  
  My kubeadm-config is:
  
  ```bash
  % kubectl describe configmap kubeadm-config -n kube-system
  Name:         kubeadm-config
  Namespace:    kube-system
  Labels:       <none>
  Annotations:  <none>
  
  Data
  ====
  ClusterConfiguration:
  ----
  apiServer:
    certSANs:
    - <my tailscale ip address>
    extraArgs:
      authorization-mode: Node,RBAC
    timeoutForControlPlane: 4m0s
  apiVersion: kubeadm.k8s.io/v1beta3
  certificatesDir: /etc/kubernetes/pki
  clusterName: kubernetes
  controlPlaneEndpoint: 192.168.0.121:6443
  controllerManager: {}
  dns: {}
  etcd:
    local:
      dataDir: /var/lib/etcd
  imageRepository: registry.k8s.io
  kind: ClusterConfiguration
  kubernetesVersion: v1.25.3
  networking:
    dnsDomain: cluster.local
    podSubnet: 10.244.0.0/16
    serviceSubnet: 10.96.0.0/12
  scheduler: {}
  
  ```
  
  which was created by
  
  `kubeadm init --control-plane-endpoint=192.168.0.121 --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=<my tailnet ip address>`
  
  Any insight would be greatly appreciated!
  
  **Tags:** kubernetes, prometheus, kubelet
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-05. Score: 1
  
  You need to send bearer token as well, like this:
  
  ```
  - job_name: 'kubelet'
          kubernetes_sd_configs:
          - role: node
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: true 
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks username_1! That changed the response from "server returned HTTP status 401 Unauthorized" to "server returned HTTP status 403 Forbidden".   However, I still don't know why it is not authorized  
  
  ---
  
  