Answers: <p>I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.</p>
  
  <p>There're several factors which I think one has to consider:</p>
  
  <ul>
  <li>How cheap/expensinve a dedicated instance is?</li>
  <li>What are security concerns/considerations?</li>
  <li>What is expected volume of data from every environment?</li>
  </ul>
  
  <p>This table can resume them :</p>
  
  <pre><code> |    Factor \ |  Solution I  | Solution II  | Solution III |
   |____________\|______________|______________|______________|
   | Data Volume |     Low      |     High     |     High     |
   |-------------+--------------+--------------+--------------|
   |    Security |      No      |     Yes      |     Yes      |
   |-------------+--------------+--------------+--------------|
   |Instance Cost|     High     |     Low      |    Medium    |
   |_____________|______________|______________|______________|
  </code></pre>
  
  <p>Based on the analysis of one's particular circumstances a preferred solution may be selected.</p>
  
  <p>I would love to know what others think about the subject, though</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-02-18T08:38:44.287Z
Answers: _"I would love to know what others think about the subject"_, yes me too.
Answers: gervais.b
Answers: CC BY-SA 4.0
Answers: 2020-02-17T15:21:13.533Z
Answers: 2020-02-18T08:41:25.990Z
Answers: 2020-02-18T08:41:25.990Z
Answers: Alexander Pushkarev
Answers: <p>You covered the obvious security issue, use HTTPS for all your connections. </p>
  
  <p>While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling out at this time ...</p>
  
  <p>Grafana with Prometheus Federated cluster consuming data from Prometheus Servers deployed in each product. Thus, my data storage needs for these 'child' clusters is minimal and no need to worry about redundancy. My Federated Cluster will be storing data in a PostgreSQL server (agents available to handle the Prometheus TSDB data are available). I could then replicate this PostgreSQL server across regions for DR with a Grafana Interface per region for redundancy. I have chosen the PostgreSQL Server option over disk volumes for ease of replication.</p>
  
  <p>As you have realized, there are several options on how to perform this task.</p>
  
  <p>Background: My use case includes multiple stand-alone SaaS products as options within a bundled solution.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-02-19T22:15:32.613Z
Answers: Thanks. We have less requirements than you but this looks like our preferred solution which is the "Solution III". I was not aware of the federation in Prometheus, that's while I was thinking to use something like https://github.com/pambrose/prometheus-proxy that push metrics in Prometheus instead of letting it pulling them. What would be the advantage of using the Federation or the proxy ?
Answers: gervais.b
Answers: CC BY-SA 4.0
Answers: 2020-02-20T00:23:16.910Z
Answers: [Prometheus Federation](https://prometheus.io/docs/prometheus/latest/federation/) allows you to effectively define parent / child relationship to allow the parent to poll data from the children into a centralized repository. Whereas, Prometheus Proxy allows you to pass your Grafana connection through a firewall to your Prometheus Server. These are entirely different concepts.
Answers: Steven K7FAQ
Answers: CC BY-SA 4.0
Answers: 2020-02-20T07:46:58.683Z
Answers: I fear that we are not speaking about the same _proxy_, the one in my link uses one _agent_ that push metrics to a _proxy_ through a firewall. So I guess that the main difference with the _federation_ is that the _"Pambrose proxy"_ push the metrics instead of letting another pulling them.
Answers: gervais.b
Answers: CC BY-SA 4.0
Answers: 2020-02-20T12:53:26.830Z
Answers: Apologies, i misread that page. Yes, you are correct, an agent gathers metrics and pushes them through a firewall to the Prometheus server.
Answers: Steven K7FAQ
Answers: CC BY-SA 4.0
Answers: 2020-02-19T21:21:18.010Z
Answers: 2020-02-19T21:21:18.010Z
Answers: Steven K7FAQ
Answers: <p>With the use case III, have you looked into <a href="https://thanos.io/" rel="nofollow noreferrer">Thanos</a> or <a href="https://cortexmetrics.io/" rel="nofollow noreferrer">Cortex</a> ?
  Both of these will provide:-</p>
  <ul>
  <li>Horizontal scalability (with deduplication)</li>
  <li>Support for secure communication with the central Thanos querier.</li>
  <li>Long term data storage support in S3 buckets and such, if required.</li>
  <li>Single place to manage all your Prometheus instances rather than adding individually into Grafana.</li>
  </ul>
  
Answers: CC BY-SA 4.0
Answers: 2021-05-25T20:48:37.867Z
Answers: 2021-05-25T20:48:37.867Z
Answers: rohatgisanat
Body: <p>I am just wondering. If I have many environments monitored via Prometheus,  what will be the best configuration? 
  The <em>security</em> and <em>efficiency</em> are already important for on-premise installations but they are much more important and unavoidable when deployed on a cloud provider.</p>
  
  <p>From your experience, what is the most common and useful configuration?
  I see three main scenarios feel free to propose something else. </p>
  
  <p>Thanks.</p>
  
  <p><strong>Solution I, Duplicate nothing</strong></p>
  
  <p>It is easy for the monitoring user because there is only one Grafana with one Datasource. However, from a security point of view, I have to open a port for each service in each environment. I will, of course, configure one reverse proxy in each environment to secure the exporters over Https and Basic authentication. </p>
  
  <pre><code>+---------+       +------------+       +- Environment A -------------+
  | Grafana |       | Prometheus |       |     +-----------+           |
  |         |------&gt;|            |--+--+------&gt;| Service 1 |           |
  +---------+       +------------+  |  | |     |           |           |
                                    |  | |     +-----------+           |
                                    |  | |                             |
                                    |  | |     +-----------+           |
                                    |  +------&gt;| Service 2 |           |
                                    |    |     |           |           |
                                    |    |     +-----------+           |
                                    |    +-----------------------------+
                                    | 
                                    |    +- Environment B -------------+
                                    |    |     +-----------+           |
                                    +---------&gt;| Service 1 |           |
                                    |    |     |           |           |
                                    |    |     +-----------+           |
                                    |    |                             |
                                    |    |     +-----------+           |
                                    +---------&gt;| Service 2 |           |
                                         |     |           |           |
                                         |     +-----------+           |
                                         +-----------------------------+
  </code></pre>
  
  <p><strong>Solution II, Duplicate everything</strong></p>
  
  <p>This looks like a popular solution and may have many benefits from a deployment point of view. I find it quite elegant because we can have on <code>docker-compose.yml</code> that set up everything. But while it can be interesting for security segregation, there will be many Grafana (each with one Prometheus data source) and that may not be useful for the monitoring user.</p>
  
  <pre><code>+- Environment A ---------------------------------------------------+
  | +---------+       +------------+                                  |
  | | Grafana |       | Prometheus |                                  |
  | |         |------&gt;|            |                                  |
  | +---------+       +------------+    +-----------+                 |
  |                             |       | Service 1 |                 |
  |                             +------&gt;|           |                 |
  |                             |       +-----------+                 |
  |                             |                                     |
  |                             |       +-----------+                 |
  |                             |       | Service 2 |                 |
  |                             +------&gt;|           |                 |
  |                                     +-----------+                 |
  +-------------------------------------------------------------------+
  </code></pre>
  
  <p><strong>Solution III, Duplicate Prometheus</strong></p>
  
  <p>This is almost the same as <strong>Solution II</strong> except that I keep one Grafana with many Prometheus data sources. This looks neat too but I still have to secure my Prometheus instance behind a reverse proxy.</p>
  
  <p>I am also thinking to create a <strong>Solution III Bis</strong> with the use of a kind of proxy or push-exporter in place of <em>Prometheus</em> in each environment. </p>
  
  <p><strong>a)</strong> The usage of a proxy does not bring much value except having only one data source in Grafana. </p>
  
  <p><strong>b)</strong> The usage of a push-exporter may remove the external entry point in each environment however I do not know how I will be "informed" when one environment is down (since it will not push anything).</p>
  
  <pre><code>                 +- Environment A ----------------------------------+
   +---------+     | +------------+                                   |
   | Grafana |     | | Prometheus |                                   |
   |         |-+----&gt;|            |                                   |
   +---------+ |   | +------------+    +-----------+                  |
               |   |           |       | Service 1 |                  |
               |   |           +------&gt;|           |                  |
               |   |           |       +-----------+                  |
               |   |           |                                      |
               |   |           |       +-----------+                  |
               |   |           |       | Service 2 |                  |
               |   |           +------&gt;|           |                  |
               |   |                   +-----------+                  |
               |   +--------------------------------------------------+
               |
               |   +- Environment B ----------------------------------+
               |   | +------------+                                   |
               |   | | Prometheus |                                   |
               +----&gt;|            |                                   |
                   | +------------+    +-----------+                  |
                   |           |       | Service 1 |                  |
                   |           +------&gt;|           |                  |
                   |           |       +-----------+                  |
                   |           |                                      |
                   |           |       +-----------+                  |
                   |           |       | Service 2 |                  |
                   |           +------&gt;|           |                  |
                   |                   +-----------+                  |
                   +--------------------------------------------------+
  </code></pre>
  
Tags: <security><monitoring><prometheus><grafana>
Title: What is the Prometheus and Grafana ideal setup?
UserDisplayName: gervais.b
ThreadText: # What is the Prometheus and Grafana ideal setup?
  
  (Asked by: username_0 on 2020-02-17)
  
  I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration?
  The *security* and *efficiency* are already important for on-premise installations but they are much more important and unavoidable when deployed on a cloud provider.
  
  From your experience, what is the most common and useful configuration?
  I see three main scenarios feel free to propose something else.
  
  Thanks.
  
  **Solution I, Duplicate nothing**
  
  It is easy for the monitoring user because there is only one Grafana with one Datasource. However, from a security point of view, I have to open a port for each service in each environment. I will, of course, configure one reverse proxy in each environment to secure the exporters over Https and Basic authentication.
  
  ```
  +---------+       +------------+       +- Environment A -------------+
  | Grafana |       | Prometheus |       |     +-----------+           |
  |         |------>|            |--+--+------>| Service 1 |           |
  +---------+       +------------+  |  | |     |           |           |
                                    |  | |     +-----------+           |
                                    |  | |                             |
                                    |  | |     +-----------+           |
                                    |  +------>| Service 2 |           |
                                    |    |     |           |           |
                                    |    |     +-----------+           |
                                    |    +-----------------------------+
                                    | 
                                    |    +- Environment B -------------+
                                    |    |     +-----------+           |
                                    +--------->| Service 1 |           |
                                    |    |     |           |           |
                                    |    |     +-----------+           |
                                    |    |                             |
                                    |    |     +-----------+           |
                                    +--------->| Service 2 |           |
                                         |     |           |           |
                                         |     +-----------+           |
                                         +-----------------------------+
  
  ```
  
  **Solution II, Duplicate everything**
  
  This looks like a popular solution and may have many benefits from a deployment point of view. I find it quite elegant because we can have on `docker-compose.yml` that set up everything. But while it can be interesting for security segregation, there will be many Grafana (each with one Prometheus data source) and that may not be useful for the monitoring user.
  
  ```
  +- Environment A ---------------------------------------------------+
  | +---------+       +------------+                                  |
  | | Grafana |       | Prometheus |                                  |
  | |         |------>|            |                                  |
  | +---------+       +------------+    +-----------+                 |
  |                             |       | Service 1 |                 |
  |                             +------>|           |                 |
  |                             |       +-----------+                 |
  |                             |                                     |
  |                             |       +-----------+                 |
  |                             |       | Service 2 |                 |
  |                             +------>|           |                 |
  |                                     +-----------+                 |
  +-------------------------------------------------------------------+
  
  ```
  
  **Solution III, Duplicate Prometheus**
  
  This is almost the same as **Solution II** except that I keep one Grafana with many Prometheus data sources. This looks neat too but I still have to secure my Prometheus instance behind a reverse proxy.
  
  I am also thinking to create a **Solution III Bis** with the use of a kind of proxy or push-exporter in place of *Prometheus* in each environment.
  
  **a)** The usage of a proxy does not bring much value except having only one data source in Grafana.
  
  **b)** The usage of a push-exporter may remove the external entry point in each environment however I do not know how I will be "informed" when one environment is down (since it will not push anything).
  
  ```
                   +- Environment A ----------------------------------+
   +---------+     | +------------+                                   |
   | Grafana |     | | Prometheus |                                   |
   |         |-+---->|            |                                   |
   +---------+ |   | +------------+    +-----------+                  |
               |   |           |       | Service 1 |                  |
               |   |           +------>|           |                  |
               |   |           |       +-----------+                  |
               |   |           |                                      |
               |   |           |       +-----------+                  |
               |   |           |       | Service 2 |                  |
               |   |           +------>|           |                  |
               |   |                   +-----------+                  |
               |   +--------------------------------------------------+
               |
               |   +- Environment B ----------------------------------+
               |   | +------------+                                   |
               |   | | Prometheus |                                   |
               +---->|            |                                   |
                   | +------------+    +-----------+                  |
                   |           |       | Service 1 |                  |
                   |           +------>|           |                  |
                   |           |       +-----------+                  |
                   |           |                                      |
                   |           |       +-----------+                  |
                   |           |       | Service 2 |                  |
                   |           +------>|           |                  |
                   |                   +-----------+                  |
                   +--------------------------------------------------+
  
  ```
  
  **Tags:** security, monitoring, prometheus, grafana
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-02-17. Score: 3
  
  I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.
  
  There're several factors which I think one has to consider:
  
  * How cheap/expensinve a dedicated instance is?
  * What are security concerns/considerations?
  * What is expected volume of data from every environment?
  
  This table can resume them :
  
  ```
   |    Factor \ |  Solution I  | Solution II  | Solution III |
   |____________\|______________|______________|______________|
   | Data Volume |     Low      |     High     |     High     |
   |-------------+--------------+--------------+--------------|
   |    Security |      No      |     Yes      |     Yes      |
   |-------------+--------------+--------------+--------------|
   |Instance Cost|     High     |     Low      |    Medium    |
   |_____________|______________|______________|______________|
  
  ```
  
  Based on the analysis of one's particular circumstances a preferred solution may be selected.
  
  I would love to know what others think about the subject, though
  
  > **Comments:**
  > **username_0**: _"I would love to know what others think about the subject"_, yes me too.  
  
  ---
  
  ### Answer by username_2 on 2020-02-19. Score: 1
  
  You covered the obvious security issue, use HTTPS for all your connections.
  
  While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling out at this time ...
  
  Grafana with Prometheus Federated cluster consuming data from Prometheus Servers deployed in each product. Thus, my data storage needs for these 'child' clusters is minimal and no need to worry about redundancy. My Federated Cluster will be storing data in a PostgreSQL server (agents available to handle the Prometheus TSDB data are available). I could then replicate this PostgreSQL server across regions for DR with a Grafana Interface per region for redundancy. I have chosen the PostgreSQL Server option over disk volumes for ease of replication.
  
  As you have realized, there are several options on how to perform this task.
  
  Background: My use case includes multiple stand-alone SaaS products as options within a bundled solution.
  
  > **Comments:**
  > **username_0**: Thanks. We have less requirements than you but this looks like our preferred solution which is the "Solution III". I was not aware of the federation in Prometheus, that's while I was thinking to use something like https://github.com/pambrose/prometheus-proxy that push metrics in Prometheus instead of letting it pulling them. What would be the advantage of using the Federation or the proxy ?  
  > **username_2**: [Prometheus Federation](https://prometheus.io/docs/prometheus/latest/federation/) allows you to effectively define parent / child relationship to allow the parent to poll data from the children into a centralized repository. Whereas, Prometheus Proxy allows you to pass your Grafana connection through a firewall to your Prometheus Server. These are entirely different concepts.  
  > **username_0**: I fear that we are not speaking about the same _proxy_, the one in my link uses one _agent_ that push metrics to a _proxy_ through a firewall. So I guess that the main difference with the _federation_ is that the _"Pambrose proxy"_ push the metrics instead of letting another pulling them.  
  > **username_2**: Apologies, i misread that page. Yes, you are correct, an agent gathers metrics and pushes them through a firewall to the Prometheus server.  
  
  ---
  
  ### Answer by username_3 on 2021-05-25. Score: 0
  
  With the use case III, have you looked into [Thanos](https://thanos.io/) or [Cortex](https://cortexmetrics.io/) ?
  Both of these will provide:-
  
  * Horizontal scalability (with deduplication)
  * Support for secure communication with the central Thanos querier.
  * Long term data storage support in S3 buckets and such, if required.
  * Single place to manage all your Prometheus instances rather than adding individually into Grafana.
  
  
  ---
  
  