Answers: <p>I don't know if you managed to get this working, but I had the exact same issue.</p>
  <p>Running a K8s cluster with <code>kube-prometheus-stack</code> installed via Helm, and a CRD prometheus-operator. I also needed to monitor an external Windows SQL VM outside the cluster, in the same network CIDR, with the K8s cluster &amp; SQL VM being in separate subnets.</p>
  <p>I ended up installing the <a href="https://github.com/prometheus-community/windows_exporter" rel="nofollow noreferrer">Windows-Exporter</a> on the SQL VM. How I got Prometheus to pick up and scrape this external target is as follows:</p>
  <ol>
  <li>I defined a ServiceMonitor.</li>
  <li>I defined a <em><a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services" rel="nofollow noreferrer">headless</a></em> Service (it has no label selector specified)</li>
  <li>I then defined a custom <a href="https://kubernetes.io/docs/reference/kubernetes-api/service-resources/endpoints-v1/" rel="nofollow noreferrer">Endpoints</a> object that the headless Service uses.</li>
  </ol>
  <p><strong>Note</strong>: Because we define a headless Service we need to manually create/manage the underlying <code>Endpoints</code>, as Services create those by default. Also, labelling your resources correctly are super important.</p>
  <p>Here are the relevant YAML files:</p>
  <ol>
  <li>ServiceMonitor (SM) yaml:</li>
  </ol>
  <pre>
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: sql-server-metrics # This name is usually important
      namespace: monitoring
      labels:
        # The below meta labels are NB! 
        # It is used to match the SVC to SM and the EP to the SVC.
        kubernetes.io/service-name: sql-server-metrics 
        app.kubernetes.io/name: sql-server-metrics
    spec:
      endpoints:
        - port: sql
          interval: 30s
          path: /metrics
          targetPort: 9182 
      # The below selector tells SM to pick up all SVC's with these meta labels
      selector:
        matchLabels:
          kubernetes.io/service-name: sql-server-metrics
          app.kubernetes.io/name: sql-server-metrics
  </pre>
  <ol start="2">
  <li>Service (SVC) yaml:</li>
  </ol>
  <pre>
    apiVersion: v1
    kind: Service
    metadata:
      name: sql-server-metrics
      namespace: monitoring
      labels:
        kubernetes.io/service-name: sql-server-metrics
        app.kubernetes.io/name: sql-server-metrics
    spec:
      # This can be one of 4 types, check the Service docs for K8s
      type: ExternalName
      # This can be the IP or FQDN/Partial-FQDN
      externalName: {IP_OR_HOSTNAME}
      ports:
        - name: sql
          protocol: TCP
          port: 9182
          targetPort: 9182
  </pre>
  <ol start="3">
  <li>Endpoints (EP) yaml:</li>
  </ol>
  <pre>
    apiVersion: v1
    kind: Endpoints
    metadata:
      name: sql-server-metrics
      namespace: monitoring
      labels:
        # This is needed to match the selector-less External Service to this Endpoint
        kubernetes.io/service-name: sql-server-metrics
        app.kubernetes.io/name: sql-server-metrics
    subsets:
      - addresses:
        - ip: {VM_IP}
          hostname: {VM_HOSTNAME}
        ports:
          - name: sql
            port: 9182
            protocol: TCP
  </pre>
  <p>I do hope the above response helps you or anyone in the future, please do feel free to mark this as the answer or to ask for any clarity and I will respond ASAP.</p>
  <p>Chow!</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-06-22T07:31:42.100Z
Answers: 2023-06-22T07:31:42.100Z
Answers: SterbenTheOG
Body: <p>I installed prometheus-operator using <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">kube-prometheus-stack</a> and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the Kubernetes cluster.</p>
  <p>I can't add the target to prometheus.yml because the storage is read-only. I saw that I could use ServiceMonitor for this purpose, but even following the steps in the official documentation, the exporter is not detected.</p>
  
Tags: <kubernetes><prometheus>
Title: How to add an aditional ServiceMonitor for prometheus-operator?
UserDisplayName: timeuz
ThreadText: # How to add an aditional ServiceMonitor for prometheus-operator?
  
  (Asked by: username_0 on 2022-12-21)
  
  I installed prometheus-operator using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the Kubernetes cluster.
  
  I can't add the target to prometheus.yml because the storage is read-only. I saw that I could use ServiceMonitor for this purpose, but even following the steps in the official documentation, the exporter is not detected.
  
  **Tags:** kubernetes, prometheus
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-22. Score: 1 | Accepted answer
  
  I don't know if you managed to get this working, but I had the exact same issue.
  
  Running a K8s cluster with `kube-prometheus-stack` installed via Helm, and a CRD prometheus-operator. I also needed to monitor an external Windows SQL VM outside the cluster, in the same network CIDR, with the K8s cluster & SQL VM being in separate subnets.
  
  I ended up installing the [Windows-Exporter](https://github.com/prometheus-community/windows_exporter) on the SQL VM. How I got Prometheus to pick up and scrape this external target is as follows:
  
  1. I defined a ServiceMonitor.
  2. I defined a *[headless](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services)* Service (it has no label selector specified)
  3. I then defined a custom [Endpoints](https://kubernetes.io/docs/reference/kubernetes-api/service-resources/endpoints-v1/) object that the headless Service uses.
  
  **Note**: Because we define a headless Service we need to manually create/manage the underlying `Endpoints`, as Services create those by default. Also, labelling your resources correctly are super important.
  
  Here are the relevant YAML files:
  
  1. ServiceMonitor (SM) yaml:
  
  ```
  
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: sql-server-metrics # This name is usually important
      namespace: monitoring
      labels:
        # The below meta labels are NB! 
        # It is used to match the SVC to SM and the EP to the SVC.
        kubernetes.io/service-name: sql-server-metrics 
        app.kubernetes.io/name: sql-server-metrics
    spec:
      endpoints:
        - port: sql
          interval: 30s
          path: /metrics
          targetPort: 9182 
      # The below selector tells SM to pick up all SVC's with these meta labels
      selector:
        matchLabels:
          kubernetes.io/service-name: sql-server-metrics
          app.kubernetes.io/name: sql-server-metrics
  
  ```
  
  2. Service (SVC) yaml:
  
  ```
  
    apiVersion: v1
    kind: Service
    metadata:
      name: sql-server-metrics
      namespace: monitoring
      labels:
        kubernetes.io/service-name: sql-server-metrics
        app.kubernetes.io/name: sql-server-metrics
    spec:
      # This can be one of 4 types, check the Service docs for K8s
      type: ExternalName
      # This can be the IP or FQDN/Partial-FQDN
      externalName: {IP_OR_HOSTNAME}
      ports:
        - name: sql
          protocol: TCP
          port: 9182
          targetPort: 9182
  
  ```
  
  3. Endpoints (EP) yaml:
  
  ```
  
    apiVersion: v1
    kind: Endpoints
    metadata:
      name: sql-server-metrics
      namespace: monitoring
      labels:
        # This is needed to match the selector-less External Service to this Endpoint
        kubernetes.io/service-name: sql-server-metrics
        app.kubernetes.io/name: sql-server-metrics
    subsets:
      - addresses:
        - ip: {VM_IP}
          hostname: {VM_HOSTNAME}
        ports:
          - name: sql
            port: 9182
            protocol: TCP
  
  ```
  
  I do hope the above response helps you or anyone in the future, please do feel free to mark this as the answer or to ask for any clarity and I will respond ASAP.
  
  Chow!
  
  
  ---
  
  