Answers: <p>Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.</p>
  <p>You also need to define the port named <code>prometheus</code> under the ingress-nginx service or deployment.</p>
  <p>ServiceMonitors can be debugged this way:-</p>
  <ol>
  <li><p><strong>Check if the ServiceMonitor created is being read by the Prometheus</strong>:-
  Look at the /targets URL. (There should be a target in 0/0
  state at least and use the old Prometheus UI) If not, that means the ServiceMonitor itself is not being picked up by Prometheus.I suggest looking into the following configuration in
  your kube-prometheus-stack configuration.</p>
  <pre><code>    serviceMonitorSelectorNilUsesHelmValues: false  # is true by default
      serviceMonitorSelector: {} #labels which help Prom identify which servicemonitors to pick up
      serviceMonitorNamespaceSelector: {} # If this is empty then all servicemonitors in any namespace will be picked up.
  </code></pre>
  <p>The default ServiceMonitor has the Helm metadata attached to it
  which is used by the Prometheus Operator to filter/choose the
  ServiceMonitors to monitor. Setting
  <code>serviceMonitorSelectorNilUsesHelmValues:false</code> will ignore any such
  selection.</p>
  </li>
  <li><p><strong>If the ServiceMonitor is visible in targets but there are no targets.</strong>:-
  In this case the issue lies between the ServiceMonitor and
  the pods it is trying to scrape.Check if the ports you mentioned are
  accessible and the pods fulfill the selectors mentioned (all <code>selectors.matchLabels</code> must match).</p>
  </li>
  </ol>
  <p>My advice would be to start another dummy ServiceMonitor by following <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus-operator-part3/" rel="nofollow noreferrer">this</a> and then modifying the ServiceMonitor one step at a time till it starts monitoring the pods</p>
  <p>Edit:-</p>
  <p>Moving all servicemonitors to the kube-prometheus-stack namespace is not a good practice. Multiple Helm charts generate their own ServiceMonitor objects which are applied in their own namespace.Enabling the prometheus operator to monitor all namespaces for servicemonitors is a better choice.</p>
  <p>For nginx side (ports and label would vary based on your implementations, update the same in the yaml below.):-</p>
  <ol>
  <li><p>Add the port in the nginx deployment</p>
  <pre class="lang-yaml prettyprint-override"><code> - containerPort: 10254
     name: prometheus
     protocol: TCP
  </code></pre>
  </li>
  <li><p>Create a service to map to the monitoring port.</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Service
  metadata:
    labels: [] # labels of your choice
    name: {} # correct name
  spec:
    clusterIP: &quot;&quot;
    ports:
      - name: prometheus
        port: 9913
        targetPort: prometheus
    selector: &lt;all labels in your pods should match&gt;
    type: ClusterIP
  
  </code></pre>
  </li>
  <li><p>Create the ServiceMonitor</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
      annotations: [] # as required.
      name: ingress-nginx-monitoring
  spec:
    endpoints:
    - interval: 30s
      path: /metrics
      port: prometheus
    namespaceSelector:
      matchNames:
      - &lt;namespace where nginx is deployed&gt;
    selector:
      matchLabels: [] # all labels in the given in the service above should match.
  </code></pre>
  </li>
  </ol>
  <pre><code>
  </code></pre>
  
Answers: CC BY-SA 4.0
Answers: 2021-06-07T15:17:10.237Z
Answers: I do see the ServiceMonitor being discovered within the Prometheus UI, but I see 0/8 active targets. I do have a port named `prometheus` going to `9113` on the `ingress-nginx-controller`. I'm not sure if that's the right port or if `ingress-nginx` even has the metrics exposed to begin with.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T15:49:20.963Z
Answers: I've done some more digging, check my edits.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T17:59:28.180Z
Answers: If I move the `ServiceMonitor` out of the `kube-prometheus-stack` namespace, I can no longer see it in the Prometheus UI. I'd rather keep all my resources in a namespace if possible anyway.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T18:07:32.550Z
Answers: That should be okay. The rest of the steps would remain the same.
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T18:20:17.757Z
Answers: I don't see the `ServiceMonitor` showing up again in Prometheus when I move it back to the `kube-prometheus-stack` namespace, it's totally gone again. I created a `ClusterIP` service to expose the prometheus port on the controller deployment but that doesn't seem to be finding it either, and I tested that it works by port-fowarding the service and I can confirm that I do see the metrics displayed at `/metrics`.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T18:30:30.233Z
Answers: What changes did you make to the ServiceMonitor? If the service works correctly and you can see the metrics, then the problem lies in the service monitor. Can you confirm if the matchLabels are correct? They should be equal to the labels present in the Service.
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T18:46:51.180Z
Answers: Okay, so connecting via the service WAS working a short while ago, now port-forwarding to it is just timing out after changing the namespace back. I can port-forward directly to the controller deployment on port 10254 though and that does work. My labels all look correct to me.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T19:06:38.093Z
Answers: Can you share the results of ‘kubectl get ep’?
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T19:07:51.833Z
Answers: My bad, I've got the service working now, but I'm just back to where I was before. I can see the monitor in Prometheus but 0/8 active targets still.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T19:21:35.333Z
Answers: If the serviceMonitor is the same as in the question and the matchLabels are all equal to the service and the matchNames has the name space then you can try adding the path: /metrics under the port. If that does not work you can create a new serviceMonitor from the template i provided. Or paste the one you are using along with prometheus screenshot.
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T19:35:57.163Z
Answers: I edited my post with more details
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:18:18.037Z
Answers: Both the configs work on my end. Only thing I had to change was the selectors in the Service. Can you get the output of ‘kubectl get ep’ also can you check what it says under the /target UI? It is even monitoring the nginx pods?
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:25:06.310Z
Answers: That command gives me one endpoint with the name `kubernetes`, and an IP with port 443. In the /target UI, nginx is not in the list.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:34:21.703Z
Answers: kubectl get ep -n ingress-nginx should give you the name of the service along with Endpoint attached to it. Under the /targets UI the ingress-nginx-metrics should have 0/8. Under that you should see URL and other details related to scraping
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:34:28.683Z
Answers: Okay, I didn't realize I needed to specify the namespace with the command. When using the correct namespace, I do see the `ingress-nginx-metrics` service, but it has no IP assigned to it. I only see the 0/8 targets on the /service-discovery page. nginx doesn't show at all on /targets.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:37:53.530Z
Answers: Yes, If you mean under the endpoint column, that means your service in not connected to any pods. Check if the selector field is correct. Selectors should match the labels on the pods.
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:43:27.703Z
Answers: I checked it over and over again, It's definitely right. I copied that YAML in my post edit directly from the dashboard. The metrics service has the `app: ingress-nginx-controller` selector, and the nginx controller deployment and service both have the label `app: ingress-nginx-controller`.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:49:20.213Z
Answers: ‘’’app.kubernetes.io/component: controller
  app.kubernetes.io/instance: ingress-nginx
   app.kubernetes.io/name: ingress-nginx’’’
  Try adding these three to the service.selector
Answers: rohatgisanat
Answers: CC BY-SA 4.0
Answers: 2021-06-07T20:53:13.260Z
Answers: I figured it out actually. I only had the label set on the Deployment, not on the Pod template. It's working now!
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-05T10:52:12.890Z
Answers: 2021-06-07T17:06:05.140Z
Answers: 2021-06-07T17:06:05.140Z
Answers: rohatgisanat
Answers: <p>The service monitor seems right, only thing is that the service monitor should be in the namespace of the Prometheus stack.</p>
  <p>Try the below service monitor:</p>
  <pre><code>apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app: ingress-nginx
      release: kube-prometheus-stack
    name: ingress-nginx-metrics
    namespace: kube-prometheus-stack
  spec:
    endpoints:
      - interval: 30s
        port: prometheus
    namespaceSelector:
      matchNames:
        - ingress-nginx
    selector:
      matchLabels:
        app: ingress-nginx
  </code></pre>
  <p>Also when you do, try port-forwarding the Prometheus</p>
  <pre><code> kubectl port-forward svc/&lt;SERVICE_NAME_OF_PROMETHEUS&gt; -n kube-prometheus-stack 9090:9090
  </code></pre>
  <p>and check in <code>Targets</code> and <code>Service Discovery</code> options in Prometheus.
  <a href="https://i.sstatic.net/6vO8v.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6vO8v.png" alt="enter image description here" /></a>
  You can see the service monitor in <code>Service Discovery</code> and exposed metrics in <code>Targets</code>.</p>
  
Answers: CC BY-SA 4.0
Answers: 2021-06-07T14:24:52.257Z
Answers: Okay so I moved it to the `kube-prometheus-stack` namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the `/metrics` path in the endpoint? I've tried with and without and it doesn't seem to change anything.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-07T15:49:24.417Z
Answers: I've done some more digging, check my edits.
Answers: chrispytoes
Answers: CC BY-SA 4.0
Answers: 2021-06-05T18:33:34.960Z
Answers: 2021-06-05T18:33:34.960Z
Answers: rksachin
Body: <p>A few weeks ago, I installed <code>ingress-nginx</code> in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my <code>ingress-nginx-controller</code> is all set up and working, but I am unable to change settings using Helm to create the <code>ServiceMonitor</code> and scrape metrics from it.</p>
  <p>I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor <code>ingress-nginx</code> without using Helm. I understand I need to make a <code>ServiceMonitor</code> resource, but I can't figure out how to make one specifically for <code>ingress-nginx</code> because every solution I find online pretty much boils down to: &quot;oh just use Helm bro&quot;.</p>
  <p>I attempted to create my own <code>ServiceMonitor</code> resource like so:</p>
  <pre><code>apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app: ingress-nginx
      release: kube-prometheus-stack
    name: ingress-nginx-metrics
    namespace: ingress-nginx
  spec:
    endpoints:
      - interval: 30s
        port: prometheus
    namespaceSelector:
      matchNames:
        - ingress-nginx
    selector:
      matchLabels:
        app: ingress-nginx-controller
  </code></pre>
  <p>It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my <code>ServiceMonitor</code> need to be in the <code>kube-prometheus-stack</code> namespace, the namespace of the service being monitored, or does it not matter?</p>
  <ul>
  <li>My <code>ingress-nginx-controller</code> deployment is in the <code>ingress-nginx</code> namespace.</li>
  <li><code>kube-prometheus-stack</code> is installed in the <code>kube-prometheus-stack</code> namespace.</li>
  </ul>
  <p>Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?</p>
  <p>Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the <code>/metrics</code> path in the endpoint? I've tried with and without and it doesn't seem to change anything.</p>
  <p>Edit: I created a <code>ClusterIP</code> service pointing to the <code>prometheus</code> port on my <code>ingress-nginx-controller</code> and I am able to port forward that and see the <code>/metrics</code> route which shows the Prometheus logs. Pointing my <code>ServiceMonitor</code> directly at this new <code>ClusterIP</code> service doesn't seem to be doing anything either.</p>
  <p>Do I need to point the <code>ServiceMonitor</code> directly at the <code>ingress-nginx-controller</code> deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see <code>ingress-nginx</code> in the Prometheus Service Discovery screen, but 0/8 active targets.</p>
  <p>Edit:
  Here's what I have in my resources now:</p>
  <p>NGINX Controller (relevant items only):</p>
  <pre><code>kind: Deployment
  apiVersion: apps/v1
  metadata:
    name: ingress-nginx-controller
    namespace: ingress-nginx
    labels:
      app: ingress-nginx-controller
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/version: 0.44.0
      helm.sh/chart: ingress-nginx-3.23.0
    annotations:
      prometheus.io/port: '10254'
      prometheus.io/scrape: 'true'
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    template:
      metadata:
        labels:
          app.kubernetes.io/component: controller
          app.kubernetes.io/instance: ingress-nginx
          app.kubernetes.io/name: ingress-nginx
        annotations:
          prometheus.io/port: '10254'
          prometheus.io/scrape: 'true'
      spec:
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
          - name: webhook
            containerPort: 8443
            protocol: TCP
          - name: prometheus
            containerPort: 10254
            protocol: TCP
  </code></pre>
  <p>NGINX metrics service:</p>
  <pre><code>kind: Service
  apiVersion: v1
  metadata:
    name: ingress-nginx-metrics
    namespace: ingress-nginx
    labels:
      app: ingress-nginx-metrics
  spec:
    ports:
      - name: prometheus
        protocol: TCP
        port: 10254
        targetPort: prometheus
    selector:
      app: ingress-nginx-controller
    type: ClusterIP
  </code></pre>
  <p>NGINX Service Monitor:</p>
  <pre><code>apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: ingress-nginx-metrics
    namespace: kube-prometheus-stack
    labels:
      app: ingress-nginx-metrics
      release: kube-prometheus-stack
  spec:
    endpoints:
      - interval: 30s
        path: /metrics
        port: prometheus
    namespaceSelector:
      matchNames:
        - ingress-nginx
    selector:
      matchLabels:
        app: ingress-nginx-metrics
  </code></pre>
  <p>I cannot port-forward to the metrics service I created, the connection times out.</p>
  <p>This is what I'm seeing in Prometheus:</p>
  <p><a href="https://i.sstatic.net/KEzeJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KEzeJ.png" alt="enter image description here" /></a></p>
  
Tags: <kubernetes><prometheus><ingress>
Title: Enable ingress-nginx Prometheus monitoring without Helm
UserDisplayName: chrispytoes
ThreadText: # Enable ingress-nginx Prometheus monitoring without Helm
  
  (Asked by: username_0 on 2021-06-04)
  
  A few weeks ago, I installed `ingress-nginx` in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my `ingress-nginx-controller` is all set up and working, but I am unable to change settings using Helm to create the `ServiceMonitor` and scrape metrics from it.
  
  I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor `ingress-nginx` without using Helm. I understand I need to make a `ServiceMonitor` resource, but I can't figure out how to make one specifically for `ingress-nginx` because every solution I find online pretty much boils down to: "oh just use Helm bro".
  
  I attempted to create my own `ServiceMonitor` resource like so:
  
  ```
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app: ingress-nginx
      release: kube-prometheus-stack
    name: ingress-nginx-metrics
    namespace: ingress-nginx
  spec:
    endpoints:
      - interval: 30s
        port: prometheus
    namespaceSelector:
      matchNames:
        - ingress-nginx
    selector:
      matchLabels:
        app: ingress-nginx-controller
  
  ```
  
  It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my `ServiceMonitor` need to be in the `kube-prometheus-stack` namespace, the namespace of the service being monitored, or does it not matter?
  
  * My `ingress-nginx-controller` deployment is in the `ingress-nginx` namespace.
  * `kube-prometheus-stack` is installed in the `kube-prometheus-stack` namespace.
  
  Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?
  
  Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the `/metrics` path in the endpoint? I've tried with and without and it doesn't seem to change anything.
  
  Edit: I created a `ClusterIP` service pointing to the `prometheus` port on my `ingress-nginx-controller` and I am able to port forward that and see the `/metrics` route which shows the Prometheus logs. Pointing my `ServiceMonitor` directly at this new `ClusterIP` service doesn't seem to be doing anything either.
  
  Do I need to point the `ServiceMonitor` directly at the `ingress-nginx-controller` deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see `ingress-nginx` in the Prometheus Service Discovery screen, but 0/8 active targets.
  
  Edit:
  Here's what I have in my resources now:
  
  NGINX Controller (relevant items only):
  
  ```
  kind: Deployment
  apiVersion: apps/v1
  metadata:
    name: ingress-nginx-controller
    namespace: ingress-nginx
    labels:
      app: ingress-nginx-controller
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/version: 0.44.0
      helm.sh/chart: ingress-nginx-3.23.0
    annotations:
      prometheus.io/port: '10254'
      prometheus.io/scrape: 'true'
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    template:
      metadata:
        labels:
          app.kubernetes.io/component: controller
          app.kubernetes.io/instance: ingress-nginx
          app.kubernetes.io/name: ingress-nginx
        annotations:
          prometheus.io/port: '10254'
          prometheus.io/scrape: 'true'
      spec:
        ports:
          - name: http
            containerPort: 80
            protocol: TCP
          - name: https
            containerPort: 443
            protocol: TCP
          - name: webhook
            containerPort: 8443
            protocol: TCP
          - name: prometheus
            containerPort: 10254
            protocol: TCP
  
  ```
  
  NGINX metrics service:
  
  ```
  kind: Service
  apiVersion: v1
  metadata:
    name: ingress-nginx-metrics
    namespace: ingress-nginx
    labels:
      app: ingress-nginx-metrics
  spec:
    ports:
      - name: prometheus
        protocol: TCP
        port: 10254
        targetPort: prometheus
    selector:
      app: ingress-nginx-controller
    type: ClusterIP
  
  ```
  
  NGINX Service Monitor:
  
  ```
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: ingress-nginx-metrics
    namespace: kube-prometheus-stack
    labels:
      app: ingress-nginx-metrics
      release: kube-prometheus-stack
  spec:
    endpoints:
      - interval: 30s
        path: /metrics
        port: prometheus
    namespaceSelector:
      matchNames:
        - ingress-nginx
    selector:
      matchLabels:
        app: ingress-nginx-metrics
  
  ```
  
  I cannot port-forward to the metrics service I created, the connection times out.
  
  This is what I'm seeing in Prometheus:
  
  [![enter image description here](https://i.sstatic.net/KEzeJ.png)](https://i.sstatic.net/KEzeJ.png)
  
  **Tags:** kubernetes, prometheus, ingress
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-06-05. Score: 2 | Accepted answer
  
  Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.
  
  You also need to define the port named `prometheus` under the ingress-nginx service or deployment.
  
  ServiceMonitors can be debugged this way:-
  
  1. **Check if the ServiceMonitor created is being read by the Prometheus**:-
     Look at the /targets URL. (There should be a target in 0/0
     state at least and use the old Prometheus UI) If not, that means the ServiceMonitor itself is not being picked up by Prometheus.I suggest looking into the following configuration in
     your kube-prometheus-stack configuration.
  
     ```
         serviceMonitorSelectorNilUsesHelmValues: false  # is true by default
         serviceMonitorSelector: {} #labels which help Prom identify which servicemonitors to pick up
         serviceMonitorNamespaceSelector: {} # If this is empty then all servicemonitors in any namespace will be picked up.
  
     ```
  
     The default ServiceMonitor has the Helm metadata attached to it
     which is used by the Prometheus Operator to filter/choose the
     ServiceMonitors to monitor. Setting
     `serviceMonitorSelectorNilUsesHelmValues:false` will ignore any such
     selection.
  2. **If the ServiceMonitor is visible in targets but there are no targets.**:-
     In this case the issue lies between the ServiceMonitor and
     the pods it is trying to scrape.Check if the ports you mentioned are
     accessible and the pods fulfill the selectors mentioned (all `selectors.matchLabels` must match).
  
  My advice would be to start another dummy ServiceMonitor by following [this](https://sysdig.com/blog/kubernetes-monitoring-prometheus-operator-part3/) and then modifying the ServiceMonitor one step at a time till it starts monitoring the pods
  
  Edit:-
  
  Moving all servicemonitors to the kube-prometheus-stack namespace is not a good practice. Multiple Helm charts generate their own ServiceMonitor objects which are applied in their own namespace.Enabling the prometheus operator to monitor all namespaces for servicemonitors is a better choice.
  
  For nginx side (ports and label would vary based on your implementations, update the same in the yaml below.):-
  
  1. Add the port in the nginx deployment
  
     ```yaml
      - containerPort: 10254
        name: prometheus
        protocol: TCP
  
     ```
  2. Create a service to map to the monitoring port.
  
     ```yaml
     apiVersion: v1
     kind: Service
     metadata:
       labels: [] # labels of your choice
       name: {} # correct name
     spec:
       clusterIP: ""
       ports:
         - name: prometheus
           port: 9913
           targetPort: prometheus
       selector: <all labels in your pods should match>
       type: ClusterIP
  
     ```
  3. Create the ServiceMonitor
  
     ```yaml
     apiVersion: monitoring.coreos.com/v1
     kind: ServiceMonitor
     metadata:
         annotations: [] # as required.
         name: ingress-nginx-monitoring
     spec:
       endpoints:
       - interval: 30s
         path: /metrics
         port: prometheus
       namespaceSelector:
         matchNames:
         - <namespace where nginx is deployed>
       selector:
         matchLabels: [] # all labels in the given in the service above should match.
  
     ```
  
  ```
  
  ```
  
  > **Comments:**
  > **username_0**: I do see the ServiceMonitor being discovered within the Prometheus UI, but I see 0/8 active targets. I do have a port named `prometheus` going to `9113` on the `ingress-nginx-controller`. I'm not sure if that's the right port or if `ingress-nginx` even has the metrics exposed to begin with.  
  > **username_0**: I've done some more digging, check my edits.  
  > **username_0**: If I move the `ServiceMonitor` out of the `kube-prometheus-stack` namespace, I can no longer see it in the Prometheus UI. I'd rather keep all my resources in a namespace if possible anyway.  
  > **username_1**: That should be okay. The rest of the steps would remain the same.  
  > **username_0**: I don't see the `ServiceMonitor` showing up again in Prometheus when I move it back to the `kube-prometheus-stack` namespace, it's totally gone again. I created a `ClusterIP` service to expose the prometheus port on the controller deployment but that doesn't seem to be finding it either, and I tested that it works by port-fowarding the service and I can confirm that I do see the metrics displayed at `/metrics`.  
  > **username_1**: What changes did you make to the ServiceMonitor? If the service works correctly and you can see the metrics, then the problem lies in the service monitor. Can you confirm if the matchLabels are correct? They should be equal to the labels present in the Service.  
  > **username_0**: Okay, so connecting via the service WAS working a short while ago, now port-forwarding to it is just timing out after changing the namespace back. I can port-forward directly to the controller deployment on port 10254 though and that does work. My labels all look correct to me.  
  > **username_1**: Can you share the results of ‘kubectl get ep’?  
  > **username_0**: My bad, I've got the service working now, but I'm just back to where I was before. I can see the monitor in Prometheus but 0/8 active targets still.  
  > **username_1**: If the serviceMonitor is the same as in the question and the matchLabels are all equal to the service and the matchNames has the name space then you can try adding the path: /metrics under the port. If that does not work you can create a new serviceMonitor from the template i provided. Or paste the one you are using along with prometheus screenshot.  
  > **username_0**: I edited my post with more details  
  > **username_1**: Both the configs work on my end. Only thing I had to change was the selectors in the Service. Can you get the output of ‘kubectl get ep’ also can you check what it says under the /target UI? It is even monitoring the nginx pods?  
  > **username_0**: That command gives me one endpoint with the name `kubernetes`, and an IP with port 443. In the /target UI, nginx is not in the list.  
  > **username_1**: kubectl get ep -n ingress-nginx should give you the name of the service along with Endpoint attached to it. Under the /targets UI the ingress-nginx-metrics should have 0/8. Under that you should see URL and other details related to scraping  
  > **username_0**: Okay, I didn't realize I needed to specify the namespace with the command. When using the correct namespace, I do see the `ingress-nginx-metrics` service, but it has no IP assigned to it. I only see the 0/8 targets on the /service-discovery page. nginx doesn't show at all on /targets.  
  > **username_1**: Yes, If you mean under the endpoint column, that means your service in not connected to any pods. Check if the selector field is correct. Selectors should match the labels on the pods.  
  > **username_0**: I checked it over and over again, It's definitely right. I copied that YAML in my post edit directly from the dashboard. The metrics service has the `app: ingress-nginx-controller` selector, and the nginx controller deployment and service both have the label `app: ingress-nginx-controller`.  
  > **username_1**: ‘’’app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx  app.kubernetes.io/name: ingress-nginx’’’ Try adding these three to the service.selector  
  > **username_0**: I figured it out actually. I only had the label set on the Deployment, not on the Pod template. It's working now!  
  
  ---
  
  ### Answer by username_2 on 2021-06-05. Score: 0
  
  The service monitor seems right, only thing is that the service monitor should be in the namespace of the Prometheus stack.
  
  Try the below service monitor:
  
  ```
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app: ingress-nginx
      release: kube-prometheus-stack
    name: ingress-nginx-metrics
    namespace: kube-prometheus-stack
  spec:
    endpoints:
      - interval: 30s
        port: prometheus
    namespaceSelector:
      matchNames:
        - ingress-nginx
    selector:
      matchLabels:
        app: ingress-nginx
  
  ```
  
  Also when you do, try port-forwarding the Prometheus
  
  ```
   kubectl port-forward svc/<SERVICE_NAME_OF_PROMETHEUS> -n kube-prometheus-stack 9090:9090
  
  ```
  
  and check in `Targets` and `Service Discovery` options in Prometheus.
  [![enter image description here](https://i.sstatic.net/6vO8v.png)](https://i.sstatic.net/6vO8v.png)
  You can see the service monitor in `Service Discovery` and exposed metrics in `Targets`.
  
  > **Comments:**
  > **username_0**: Okay so I moved it to the `kube-prometheus-stack` namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the `/metrics` path in the endpoint? I've tried with and without and it doesn't seem to change anything.  
  > **username_0**: I've done some more digging, check my edits.  
  
  ---
  
  