Stack Exchange
Aws EKS Prometheus helm failed pod CrashLoopBackOff with log "trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\"
I followed the steps here https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html
In a nutshell, I installed the Prometheus community helm chart on my EKS cluster in AWS https://github.com/prometheus-community/helm-charts.
My Kubernetes version is 1.27
The cluster node group has two t3.large nodes (8gb memory, 2vCPU)
After creation, I have:
Pods:
kubectl get pods -n prometheus

NAME                                                        READY   STATUS             RESTARTS         AGE
prometheus-alertmanager-0                            1/1     Running            0                142m
prometheus-kube-state-metrics-69b5df7dc4-kfqc2       1/1     Running            0                142m
prometheus-prometheus-node-exporter-dkksq            1/1     Running            0                142m
prometheus-prometheus-node-exporter-fzr5z            1/1     Running            0                142m
prometheus-prometheus-pushgateway-847c6f4d57-dz5wx   1/1     Running            0                142m
prometheus-server-5fbb47548d-sxm72                   1/2     CrashLoopBackOff   32 (3m58s ago)   142m

Pod logs:
 kubectl logs prometheus-server-5fbb47548d-sxm72 -n prometheus

....
level=error ts=2023-08-04T17:45:27.636671778Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"

Deployments:
kubectl get deployments -n prometheus
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-kube-state-metrics       1/1     1            1           144m
prometheus-prometheus-pushgateway   1/1     1            1           144m
prometheus-server                   0/1     1            0           144m

Describe failed deployment:
kubectl describe deployments prometheus-server -n prometheus

...
 Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded

The EBS volumes have successfully bound.
It seems that the prometheus-server pod is stuck in CrashLoopBackOff because it can't connect to itself on 127.0.0.1:9090
I have been searching for an answer all day and trying different things but can't solve the issue.
My helm values looks like this:
## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
  server:
    name: amp-iamproxy-ingest-service-account
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam:###########:role/service-role/Amazon_EventBridge_Invoke_Api_Destination_###########
server:
  remoteWrite:
    - url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/#############/api/v1/remote_write
      sigv4:
        region: eu-west-1
      queue_config:
        max_samples_per_send: 1000
        max_shards: 200
        capacity: 2500

I can't figure this out,
please help! I'm sure someone else has struggled with the same issue because I can't find much info on this. Why is it failing? What Can I do to fix it? Thanks in advance

I've had this issue in the past as well and I ended up noticing that my nodes were not joining the master.
Ensure that your nodes have joined the master and are in a ready state.

kubectl logs [pod-name] -c prometheus-server -n prometheus
level=error msg="Failed to apply configuration" err="could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for
check whether you missed the openID
