Answers: <p>It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen <a href="https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus" rel="nofollow noreferrer">kube-prometheus</a>? It's a one-shot install that will get you 90% of where you need to be for your cluster.</p>
  
  <p>The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having a Grafana per cluster.</p>
  
  <p>Edit: Getting started with kube-prometheus is a one command install. <code>kubectl apply -f manifests/</code></p>
  
Answers: CC BY-SA 3.0
Answers: 2018-04-04T12:24:45.693Z
Answers: I did a 15 minute Prometheus install from a Medium article. It was amazing. Still we have an ask to have the monitoring outside the cluster so that when the cluster goes down the monitoring is there. Is there a way to do that? Is the ROI high enough? Or do you still recommend in cluster?
Answers: David West
Answers: CC BY-SA 3.0
Answers: 2018-04-04T14:41:36.263Z
Answers: @DavidWest What will you monitor? Are you running ETCD on servers off the cluster as well? See when you lose the API servers, you are going to lose monitoring of your services eventually. This could be immediately or over time. This is because Prometheus relies on the API servers for service discovery and internally some meta_tags provided by the API servers for configuration. 
  
  You can use certain POD labels to ensure your Prometheus pod is the "last man standing". I believe the recommended approach is to run Prometheus on top of Kubernetes.
Answers: Levi
Answers: CC BY-SA 3.0
Answers: 2018-04-04T15:58:13.360Z
Answers: It will be there for capacity planning, but, more importantly, to monitor the cluster health.
Answers: David West
Answers: CC BY-SA 3.0
Answers: 2018-04-10T17:41:04.713Z
Answers: It seems to be working well in cluster. Plus, it was so easy to do it that way.
Answers: David West
Answers: CC BY-SA 4.0
Answers: 2019-02-24T13:30:37.860Z
Answers: How do you get the application metrics to Prometheus?
Answers: avi
Answers: CC BY-SA 4.0
Answers: 2019-02-24T21:56:06.763Z
Answers: @avi You should probably start a new question ;). Are you talking about custom metrics the application may provide? Like how long a certain function is taking or size of a queue? In that case you can add a JSON endpoint in the format Prometheus accepts, add a tag to your pod and then configure Prometheus to scrape that tag. Now you will see the metrics in Prometheus and be able to graph them in Grafana. If you mean container metrics like CPU and Ram, most Prometheus installations on Kubernetes will handle that for you.
Answers: Levi
Answers: CC BY-SA 4.0
Answers: 2019-02-25T07:25:53.507Z
Answers: @Levi Thanks. I asked a related question here:  https://devops.stackexchange.com/questions/6450/which-libraries-generate-observability-metrics-for-applications-in-the-cluster-t
Answers: avi
Answers: CC BY-SA 4.0
Answers: 2018-04-04T12:21:19.590Z
Answers: 2019-02-24T21:58:59.773Z
Answers: 2019-02-24T21:58:59.773Z
Answers: Levi
Body: <p>We want to start our monitoring off the cluster on a local server.  Is this practical and will it be feasible and reasonable to hook local Prometheus and Grafana installations into a Kubernetes cluster? </p>
  
Tags: <kubernetes><prometheus><grafana>
Title: Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster
UserDisplayName: David West
ThreadText: # Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster
  
  (Asked by: username_0 on 2018-04-03)
  
  We want to start our monitoring off the cluster on a local server. Is this practical and will it be feasible and reasonable to hook local Prometheus and Grafana installations into a Kubernetes cluster?
  
  **Tags:** kubernetes, prometheus, grafana
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-04-04. Score: 3 | Accepted answer
  
  It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen [kube-prometheus](https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus)? It's a one-shot install that will get you 90% of where you need to be for your cluster.
  
  The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having a Grafana per cluster.
  
  Edit: Getting started with kube-prometheus is a one command install. `kubectl apply -f manifests/`
  
  > **Comments:**
  > **username_0**: I did a 15 minute Prometheus install from a Medium article. It was amazing. Still we have an ask to have the monitoring outside the cluster so that when the cluster goes down the monitoring is there. Is there a way to do that? Is the ROI high enough? Or do you still recommend in cluster?  
  > **username_1**: @username_0 What will you monitor? Are you running ETCD on servers off the cluster as well? See when you lose the API servers, you are going to lose monitoring of your services eventually. This could be immediately or over time. This is because Prometheus relies on the API servers for service discovery and internally some meta_tags provided by the API servers for configuration.   You can use certain POD labels to ensure your Prometheus pod is the "last man standing". I believe the recommended approach is to run Prometheus on top of Kubernetes.  
  > **username_0**: It will be there for capacity planning, but, more importantly, to monitor the cluster health.  
  > **username_0**: It seems to be working well in cluster. Plus, it was so easy to do it that way.  
  > **username_2**: How do you get the application metrics to Prometheus?  
  > **username_1**: @username_2 You should probably start a new question ;). Are you talking about custom metrics the application may provide? Like how long a certain function is taking or size of a queue? In that case you can add a JSON endpoint in the format Prometheus accepts, add a tag to your pod and then configure Prometheus to scrape that tag. Now you will see the metrics in Prometheus and be able to graph them in Grafana. If you mean container metrics like CPU and Ram, most Prometheus installations on Kubernetes will handle that for you.  
  > **username_2**: @username_1 Thanks. I asked a related question here:  https://devops.stackexchange.com/questions/6450/which-libraries-generate-observability-metrics-for-applications-in-the-cluster-t  
  
  ---
  
  