title: K8S - using Prometheus to monitor another prometheus instance in secure way
body: <p>I've installed <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator" rel="nofollow noreferrer">Prometheus operator</a> <code>0.34</code> (which works as expected) on cluster <strong>A</strong> (main prom)
  Now I want to use the <a href="https://prometheus.io/docs/prometheus/latest/federation/" rel="nofollow noreferrer">federation</a> option,I mean collect metrics from other Prometheus which is located on other  K8S cluster <strong>B</strong> </p>
  
  <p><strong>Secnario:</strong></p>
  
  <blockquote>
    <ol>
    <li>have in cluster <strong>A</strong> MAIN <a href="https://github.com/coreos/prometheus-operator" rel="nofollow noreferrer">prometheus operator</a> <code>v0.34</code> <a href="https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml" rel="nofollow noreferrer">config</a></li>
    <li>I've in cluster <strong>B</strong> SLAVE <a href="https://github.com/helm/charts/tree/master/stable/prometheus" rel="nofollow noreferrer">prometheus</a> <code>2.13.1</code> <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml" rel="nofollow noreferrer">config</a></li>
    </ol>
  </blockquote>
  
  <p>Both installed successfully via helm, I can access to localhost via <code>port-forwarding</code> and see the scraping results on each cluster.</p>
  
  <p><strong>I did the following steps</strong></p>
  
  <p>Use on the operator (main cluster A)  <a href="https://github.com/coreos/prometheus-operator/blob/master/Documentation/additional-scrape-config.md" rel="nofollow noreferrer">additionalScrapeconfig</a> 
  I've added the following to the <code>values.yaml</code> file and update it via helm.</p>
  
  <pre><code>additionalScrapeConfigs:
   - job_name: 'federate'  
     honor_labels: true
     metrics_path: /federate
     params:
       match[]:
         - '{job="prometheus"}'
         - '{__name__=~"job:.*"}'
     static_configs:
       - targets:
         - 101.62.201.122:9090 # The External-IP and port from the target prometheus on Cluster B
  </code></pre>
  
  <p>I took the target like following:</p>
  
  <p>on prometheus inside <strong>cluster B</strong> (from which I want to collect the data) I use:</p>
  
  <p><code>kubectl get svc -n monitoring</code></p>
  
  <p>And get the following entries:</p>
  
  <p>Took the <code>EXTERNAL-IP</code> and put it inside the <code>additionalScrapeConfigs</code> config entry.</p>
  
  <p>Now I switch to cluster <code>A</code> and run <code>kubectl port-forward svc/mon-prometheus-operator-prometheus  9090:9090 -n monitoring</code> </p>
  
  <p>Open the browser with <code>localhost:9090</code> see the graph's and click on <code>Status</code> and there Click on <code>Targets</code> </p>
  
  <p>And see the new target with job <code>federate</code></p>
  
  <p><a href="https://i.stack.imgur.com/8oPqd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8oPqd.png" alt="enter image description here"></a></p>
  
  <p>Now my main question/gaps. (security &amp; verification) </p>
  
  <ol>
  <li>To be able to see that target <code>state</code> on green (see the pic) I configure the prometheus server in cluster <code>B</code> instead of using <code>type:NodePort</code> to use <a href="https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L835" rel="nofollow noreferrer"><code>type:LoadBalacer</code></a> which expose the metrics outside, this can be good for testing but I need to <strong>secure it</strong>, how it can be done ?
  How to make the e2e works in <strong>secure way</strong>...</li>
  </ol>
  
  <p>tls
  <a href="https://prometheus.io/docs/prometheus/1.8/configuration/configuration/#tls_config" rel="nofollow noreferrer">https://prometheus.io/docs/prometheus/1.8/configuration/configuration/#tls_config</a></p>
  
  <p>Inside <strong>cluster A</strong> (main cluster) we use certificate for out services with <code>istio</code> like following which works</p>
  
  <pre><code>tls:
     mode: SIMPLE
     privateKey: /etc/istio/oss-tls/tls.key
     serverCertificate: /etc/istio/oss-tls/tls.crt
  </code></pre>
  
  <p>I see that inside the doc there is an option to config</p>
  
  <pre><code>    additionalScrapeConfigs:
       - job_name: 'federate'  
         honor_labels: true
         metrics_path: /federate
         params:
           match[]:
             - '{job="prometheus"}'
             - '{__name__=~"job:.*"}'
         static_configs:
           - targets:
             - 101.62.201.122:9090 # The External-IP and port from the target
  #        tls_config:
  #          ca_file: /opt/certificate-authority-data.pem
  #          cert_file: /opt/client-certificate-data.pem
  #          key_file: /sfp4/client-key-data.pem
  #          insecure_skip_verify: true
  
  </code></pre>
  
  <p>But not sure which certificate I need to use inside the prometheus operator config , the certificate of the main prometheus A or the slave B?</p>