Answers: <p>I think your can do some kind of alerting on a metric <code>rate</code> with something like this:</p>
  
  <pre><code>ALERT DropInMetricsFromExporter
    IF rate(&lt;metric_name&gt;[1m]) == 0
    FOR 3m
    ANNOTATIONS {
      summary = "Rate of metrics is 0 {{ $labels.&lt;your_label&gt; }}",
      description = "Rate of metric dropped, actually: {{ $value }}%",
  }
  </code></pre>
  
  <p>The main idea is to alert whenever the metric rate is at 0 for 3 minutes, with the proper metric name and a label somewhere telling from which exporter it comes it should give you the correct information.  </p>
  
  <p>Choosing the right metric to monitor by exporter could be complex, without more insight is hard to give a better advice out of vacuum.<br>
  This <a href="https://prometheus.io/blog/2015/06/18/practical-anomaly-detection/" rel="noreferrer">blog post</a> could be an inspiration also for a more generic detection.</p>
  
Answers: CC BY-SA 3.0
Answers: 2017-08-31T09:49:18.280Z
Answers: rate calculates a rate per second for a given counter. This has nothing to do with the rate at which metrics are scraped (-> scrape_interval). Your example would alert if the given counter (metric_name) doesn't increase for 3 minutes which is probably not what the OP wants.
Answers: Johannes 'fish' Ziemke
Answers: CC BY-SA 3.0
Answers: 2017-08-31T10:10:08.320Z
Answers: @Johannes'fish'Ziemke That why I said finding the right metric could be complex. Using the `time` metric on a node exporter for example would do. If you have a better way to alert on an exporter failing, feel free to add an answer
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2017-08-31T10:13:41.773Z
Answers: @Johannes'fish'Ziemke Welcome on devops.se btw :)
Answers: Tensibai
Answers: CC BY-SA 3.0
Answers: 2017-03-01T13:04:33.890Z
Answers: 2017-03-01T13:04:33.890Z
Answers: Tensibai
Answers: <p>I'm not sure if this is the same problem you are seeing. But we saw these random gaps for Java applications that did not specifically set the GC settings. Meaning we were seeing gaps when the data collection stopped because the activity for the instance stopped while the JVM was doing a Full GC. If you are using Java this can happen. Our fix was to explicitly use the G1 garbage collector (Java 8+) with a specified limit on the length of GC activity to prevent these time gaps in the data collection.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-04-29T18:05:15.080Z
Answers: 2019-04-29T18:05:15.080Z
Answers: cwa
Answers: <p>There are a few reasons which might have caused the gap. Most likely the exporter isn't reachable in which case the <code>up</code> timeseries will be 0. You can alert on this like this (taken from <a href="https://prometheus.io/docs/alerting/rules/#templating" rel="nofollow noreferrer">https://prometheus.io/docs/alerting/rules/#templating</a>):</p>
  
  <pre><code># Alert for any instance that is unreachable for &gt;5 minutes.
  ALERT InstanceDown
    IF up == 0
    FOR 5m
    LABELS { severity = "page" }
    ANNOTATIONS {
      summary = "Instance {{ $labels.instance }} down",
      description = "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.",
    }
  </code></pre>
  
  <p>On the status page you should also see that it's down including an error message. Unfortunately there is no way to see past error but there is an issue to track this: <a href="https://github.com/prometheus/prometheus/issues/2820" rel="nofollow noreferrer">https://github.com/prometheus/prometheus/issues/2820</a></p>
  
  <p>Your Prometheus server can be also overloaded causing scraping to stop which too would explain the gaps. In that case you should see <code>Storage needs throttling. Scrapes and rule evaluations will be skipped.</code> errors in the log and increases in the <code>prometheus_target_skipped_scrapes_total</code> metrics. You should alert on that too, e.g:</p>
  
  <pre><code>ALERT PrometheusSkipsScrapes
    IF rate(prometheus_target_skipped_scrapes_total[2m]) &gt; 0
    FOR 5m
  </code></pre>
  
Answers: CC BY-SA 3.0
Answers: 2017-08-31T10:30:33.197Z
Answers: 2017-08-31T10:30:33.197Z
Answers: Johannes 'fish' Ziemke
Body: <p>I have been gradually integrating <a href="https://prometheus.io/" rel="noreferrer">Prometheus</a> into my monitoring workflows, in order to gather detailed metrics about running infrastructure.</p>
  
  <p>During this, I have noticed that I often run into a peculiar issue: sometimes an exporter that Prometheus is supposed to pull data from becomes unresponsive. Maybe because of a network misconfiguration - it is no longer accessible - or just because the exporter crashed.</p>
  
  <p>Whatever the reason it may be, I find that some of the data I expect to see in Prometheus is missing and there is nothing in the series for a certain time period. Sometimes, one exporter failing (timing out?) also seems to cause others to fail (first timeout pushed entire job above top-level timeout? just speculating).</p>
  
  <p><a href="https://i.sstatic.net/BqcdB.png" rel="noreferrer"><img src="https://i.sstatic.net/BqcdB.png" alt="Gap in data"></a></p>
  
  <p>All I see is a gap in the series, like shown in the above visualization. There is nothing in the log when this happens. Prometheus self-metrics also seem fairly barren. I have just had to resort to manually trying to replicate what Prometheus is doing and seeing where it breaks. This is irksome. There must be a better way! While I do not need realtime alerts, I at least want to be able to see that an exporter failed to deliver data. Even a boolean "hey check your data" flag would be a start.</p>
  
  <p>How do I obtain meaningful information about Prometheus failing to obtain data from exporters? How do I understand why gaps exist without having to perform a manual simulation of Prometheus data gathering? What are the sensible practices in this regard, perhaps even when extended to monitoring data collections in general, beyond Prometheus?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-08-28T06:01:27.367Z
Comments: We are also facing this issue. @Sander were you able to find the cause?
Comments: Deepak N
Comments: CC BY-SA 3.0
Comments: 2017-08-28T06:04:55.147Z
Comments: @DeepakN no, I have no useful insight to add here, unfortunately. It remains a pain point when using Prometheus.
Comments: Sander
Comments: CC BY-SA 3.0
Comments: 2017-03-01T10:41:14.730Z
Comments: Do you have any log entries or warnings/errors relevant to the problem?
Comments: kenorb
Comments: CC BY-SA 3.0
Comments: 2017-03-01T11:17:02.537Z
Comments: Nothing whatsoever, I just see a gap in the data series. The Prometheus output just has the normal regular lines about saving pending changes every few minutes and whatnot (unless I miss some hidden logs to look at).
Comments: Sander
Tags: <monitoring><prometheus>
Title: How do I troubleshoot missing data in my Prometheus database?
UserDisplayName: Sander
ThreadText: # How do I troubleshoot missing data in my Prometheus database?
  
  (Asked by: username_0 on 2017-03-01)
  
  I have been gradually integrating [Prometheus](https://prometheus.io/) into my monitoring workflows, in order to gather detailed metrics about running infrastructure.
  
  During this, I have noticed that I often run into a peculiar issue: sometimes an exporter that Prometheus is supposed to pull data from becomes unresponsive. Maybe because of a network misconfiguration - it is no longer accessible - or just because the exporter crashed.
  
  Whatever the reason it may be, I find that some of the data I expect to see in Prometheus is missing and there is nothing in the series for a certain time period. Sometimes, one exporter failing (timing out?) also seems to cause others to fail (first timeout pushed entire job above top-level timeout? just speculating).
  
  [![Gap in data](https://i.sstatic.net/BqcdB.png)](https://i.sstatic.net/BqcdB.png)
  
  All I see is a gap in the series, like shown in the above visualization. There is nothing in the log when this happens. Prometheus self-metrics also seem fairly barren. I have just had to resort to manually trying to replicate what Prometheus is doing and seeing where it breaks. This is irksome. There must be a better way! While I do not need realtime alerts, I at least want to be able to see that an exporter failed to deliver data. Even a boolean "hey check your data" flag would be a start.
  
  How do I obtain meaningful information about Prometheus failing to obtain data from exporters? How do I understand why gaps exist without having to perform a manual simulation of Prometheus data gathering? What are the sensible practices in this regard, perhaps even when extended to monitoring data collections in general, beyond Prometheus?
  
  **Tags:** monitoring, prometheus
  
  > **Comments:**
  > **username_2**: Do you have any log entries or warnings/errors relevant to the problem?  
  > **username_0**: Nothing whatsoever, I just see a gap in the data series. The Prometheus output just has the normal regular lines about saving pending changes every few minutes and whatnot (unless I miss some hidden logs to look at).  
  > **username_1**: We are also facing this issue. @username_0 were you able to find the cause?  
  > **username_0**: @username_1 no, I have no useful insight to add here, unfortunately. It remains a pain point when using Prometheus.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-03-01. Score: 5
  
  I think your can do some kind of alerting on a metric `rate` with something like this:
  
  ```
  ALERT DropInMetricsFromExporter
    IF rate(<metric_name>[1m]) == 0
    FOR 3m
    ANNOTATIONS {
      summary = "Rate of metrics is 0 {{ $labels.<your_label> }}",
      description = "Rate of metric dropped, actually: {{ $value }}%",
  }
  
  ```
  
  The main idea is to alert whenever the metric rate is at 0 for 3 minutes, with the proper metric name and a label somewhere telling from which exporter it comes it should give you the correct information.
  
  Choosing the right metric to monitor by exporter could be complex, without more insight is hard to give a better advice out of vacuum.  
  This [blog post](https://prometheus.io/blog/2015/06/18/practical-anomaly-detection/) could be an inspiration also for a more generic detection.
  
  > **Comments:**
  > **username_4**: rate calculates a rate per second for a given counter. This has nothing to do with the rate at which metrics are scraped (-> scrape_interval). Your example would alert if the given counter (metric_name) doesn't increase for 3 minutes which is probably not what the OP wants.  
  > **username_3**: @username_4 That why I said finding the right metric could be complex. Using the `time` metric on a node exporter for example would do. If you have a better way to alert on an exporter failing, feel free to add an answer  
  > **username_3**: @username_4 Welcome on devops.se btw :)  
  
  ---
  
  ### Answer by username_5 on 2019-04-29. Score: 1
  
  I'm not sure if this is the same problem you are seeing. But we saw these random gaps for Java applications that did not specifically set the GC settings. Meaning we were seeing gaps when the data collection stopped because the activity for the instance stopped while the JVM was doing a Full GC. If you are using Java this can happen. Our fix was to explicitly use the G1 garbage collector (Java 8+) with a specified limit on the length of GC activity to prevent these time gaps in the data collection.
  
  
  ---
  
  ### Answer by username_4 on 2017-08-31. Score: 1
  
  There are a few reasons which might have caused the gap. Most likely the exporter isn't reachable in which case the `up` timeseries will be 0. You can alert on this like this (taken from <https://prometheus.io/docs/alerting/rules/#templating>):
  
  ```
  # Alert for any instance that is unreachable for >5 minutes.
  ALERT InstanceDown
    IF up == 0
    FOR 5m
    LABELS { severity = "page" }
    ANNOTATIONS {
      summary = "Instance {{ $labels.instance }} down",
      description = "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.",
    }
  
  ```
  
  On the status page you should also see that it's down including an error message. Unfortunately there is no way to see past error but there is an issue to track this: <https://github.com/prometheus/prometheus/issues/2820>
  
  Your Prometheus server can be also overloaded causing scraping to stop which too would explain the gaps. In that case you should see `Storage needs throttling. Scrapes and rule evaluations will be skipped.` errors in the log and increases in the `prometheus_target_skipped_scrapes_total` metrics. You should alert on that too, e.g:
  
  ```
  ALERT PrometheusSkipsScrapes
    IF rate(prometheus_target_skipped_scrapes_total[2m]) > 0
    FOR 5m
  
  ```
  
  
  ---
  
  