Answers: <p>kubectl logs [pod-name] -c prometheus-server -n prometheus</p>
  <p>level=error msg=&quot;Failed to apply configuration&quot; err=&quot;could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for</p>
  <p>check whether you missed the openID</p>
  
Answers: CC BY-SA 4.0
Answers: 2024-03-14T06:25:32.837Z
Answers: 2024-03-14T06:25:32.837Z
Answers: easy.yan
Answers: <p>I've had this issue in the past as well and I ended up noticing that my nodes were not joining the master.</p>
  <p>Ensure that your nodes have joined the master and are in a ready state.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-08-08T01:36:50.967Z
Answers: 2023-08-08T04:53:53.653Z
Answers: 2023-08-08T04:53:53.653Z
Answers: Freddy
Body: <p>I followed the steps here <a href="https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html</a></p>
  <p>In a nutshell, I installed the Prometheus community helm chart on my EKS cluster in AWS <a href="https://github.com/prometheus-community/helm-charts" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts</a>.</p>
  <p>My Kubernetes version is 1.27</p>
  <p>The cluster node group has two t3.large nodes (8gb memory, 2vCPU)</p>
  <p>After creation, I have:</p>
  <p>Pods:</p>
  <pre><code>kubectl get pods -n prometheus
  
  NAME                                                        READY   STATUS             RESTARTS         AGE
  prometheus-alertmanager-0                            1/1     Running            0                142m
  prometheus-kube-state-metrics-69b5df7dc4-kfqc2       1/1     Running            0                142m
  prometheus-prometheus-node-exporter-dkksq            1/1     Running            0                142m
  prometheus-prometheus-node-exporter-fzr5z            1/1     Running            0                142m
  prometheus-prometheus-pushgateway-847c6f4d57-dz5wx   1/1     Running            0                142m
  prometheus-server-5fbb47548d-sxm72                   1/2     CrashLoopBackOff   32 (3m58s ago)   142m
  </code></pre>
  <p>Pod logs:</p>
  <pre><code> kubectl logs prometheus-server-5fbb47548d-sxm72 -n prometheus
  
  ....
  level=error ts=2023-08-04T17:45:27.636671778Z caller=runutil.go:100 msg=&quot;function failed. Retrying in next tick&quot; err=&quot;trigger reload: reload request failed: Post \&quot;http://127.0.0.1:9090/-/reload\&quot;: dial tcp 127.0.0.1:9090: connect: connection refused&quot;
  </code></pre>
  <p>Deployments:</p>
  <pre><code>kubectl get deployments -n prometheus
  NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
  prometheus-kube-state-metrics       1/1     1            1           144m
  prometheus-prometheus-pushgateway   1/1     1            1           144m
  prometheus-server                   0/1     1            0           144m
  </code></pre>
  <p>Describe failed deployment:</p>
  <pre><code>kubectl describe deployments prometheus-server -n prometheus
  
  ...
   Type           Status  Reason
    ----           ------  ------
    Available      False   MinimumReplicasUnavailable
    Progressing    False   ProgressDeadlineExceeded
  </code></pre>
  <p>The EBS volumes have successfully bound.</p>
  <p>It seems that the prometheus-server pod is stuck in CrashLoopBackOff because it can't connect to itself on 127.0.0.1:9090</p>
  <p>I have been searching for an answer all day and trying different things but can't solve the issue.</p>
  <p>My helm values looks like this:</p>
  <pre><code>## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
  ## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
  ##
  serviceAccounts:
    server:
      name: amp-iamproxy-ingest-service-account
      annotations: 
        eks.amazonaws.com/role-arn: arn:aws:iam:###########:role/service-role/Amazon_EventBridge_Invoke_Api_Destination_###########
  server:
    remoteWrite:
      - url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/#############/api/v1/remote_write
        sigv4:
          region: eu-west-1
        queue_config:
          max_samples_per_send: 1000
          max_shards: 200
          capacity: 2500
  </code></pre>
  <p>I can't figure this out,
  please help! I'm sure someone else has struggled with the same issue because I can't find much info on this. Why is it failing? What Can I do to fix it? Thanks in advance</p>
  
Tags: <kubernetes><amazon-web-services><prometheus><kubernetes-helm>
Title: Aws EKS Prometheus helm failed pod CrashLoopBackOff with log "trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\"
UserDisplayName: Harry
ThreadText: # Aws EKS Prometheus helm failed pod CrashLoopBackOff with log "trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\"
  
  (Asked by: username_0 on 2023-08-04)
  
  I followed the steps here <https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html>
  
  In a nutshell, I installed the Prometheus community helm chart on my EKS cluster in AWS <https://github.com/prometheus-community/helm-charts>.
  
  My Kubernetes version is 1.27
  
  The cluster node group has two t3.large nodes (8gb memory, 2vCPU)
  
  After creation, I have:
  
  Pods:
  
  ```
  kubectl get pods -n prometheus
  
  NAME                                                        READY   STATUS             RESTARTS         AGE
  prometheus-alertmanager-0                            1/1     Running            0                142m
  prometheus-kube-state-metrics-69b5df7dc4-kfqc2       1/1     Running            0                142m
  prometheus-prometheus-node-exporter-dkksq            1/1     Running            0                142m
  prometheus-prometheus-node-exporter-fzr5z            1/1     Running            0                142m
  prometheus-prometheus-pushgateway-847c6f4d57-dz5wx   1/1     Running            0                142m
  prometheus-server-5fbb47548d-sxm72                   1/2     CrashLoopBackOff   32 (3m58s ago)   142m
  
  ```
  
  Pod logs:
  
  ```
   kubectl logs prometheus-server-5fbb47548d-sxm72 -n prometheus
  
  ....
  level=error ts=2023-08-04T17:45:27.636671778Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
  
  ```
  
  Deployments:
  
  ```
  kubectl get deployments -n prometheus
  NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
  prometheus-kube-state-metrics       1/1     1            1           144m
  prometheus-prometheus-pushgateway   1/1     1            1           144m
  prometheus-server                   0/1     1            0           144m
  
  ```
  
  Describe failed deployment:
  
  ```
  kubectl describe deployments prometheus-server -n prometheus
  
  ...
   Type           Status  Reason
    ----           ------  ------
    Available      False   MinimumReplicasUnavailable
    Progressing    False   ProgressDeadlineExceeded
  
  ```
  
  The EBS volumes have successfully bound.
  
  It seems that the prometheus-server pod is stuck in CrashLoopBackOff because it can't connect to itself on 127.0.0.1:9090
  
  I have been searching for an answer all day and trying different things but can't solve the issue.
  
  My helm values looks like this:
  
  ```
  ## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
  ## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
  ##
  serviceAccounts:
    server:
      name: amp-iamproxy-ingest-service-account
      annotations: 
        eks.amazonaws.com/role-arn: arn:aws:iam:###########:role/service-role/Amazon_EventBridge_Invoke_Api_Destination_###########
  server:
    remoteWrite:
      - url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/#############/api/v1/remote_write
        sigv4:
          region: eu-west-1
        queue_config:
          max_samples_per_send: 1000
          max_shards: 200
          capacity: 2500
  
  ```
  
  I can't figure this out,
  please help! I'm sure someone else has struggled with the same issue because I can't find much info on this. Why is it failing? What Can I do to fix it? Thanks in advance
  
  **Tags:** kubernetes, amazon-web-services, prometheus, kubernetes-helm
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-03-14. Score: 0
  
  kubectl logs [pod-name] -c prometheus-server -n prometheus
  
  level=error msg="Failed to apply configuration" err="could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for
  
  check whether you missed the openID
  
  
  ---
  
  ### Answer by username_2 on 2023-08-08. Score: 0
  
  I've had this issue in the past as well and I ended up noticing that my nodes were not joining the master.
  
  Ensure that your nodes have joined the master and are in a ready state.
  
  
  ---
  
  