Answers: <p>I think your need may be an anti-pattern but I'm unsure.</p>
  <p>Metrics are used to measure things and Metrics are correlated with time.</p>
  <p>In your example, the job's output is a constant and not correlated with a time.</p>
  <p>Metrics are often (!) measurements of the health (of the state) of a system rather than the output (product) of a system.</p>
  <p>The job's duration, CPU, memory, success|failure etc. are conventional (!) measurements.</p>
  <p>While it's entirely reasonable to want to capture time-series data (from Jobs), it may be (!) that a database or some other persistence mechanism would be a better sink for your data.</p>
  <p>Answering your question as stated: Jobs are challenging because they are potentially unscrapeable (because they don't live sufficiently long) by Prometheus as part of its 'pull' mechanism.</p>
  <p>Batch jobs are a valid use-case for <a href="https://prometheus.io/docs/practices/pushing/" rel="nofollow noreferrer">Pushgateway</a>. So, your Jobs could push metrics to Pushgateway to ensure that these are captured.</p>
  <p>If the logs produced by your Jobs are persisted beyond the life of the Job, another approach is to derive metrics (by parsing) logs. This approach biases towards aggregation of log data. The examplar is counting HTTP 500s in log entries to determine a failure rate.</p>
  
Answers: CC BY-SA 4.0
Answers: 2022-08-09T04:22:00.550Z
Answers: Thanks Dan, I agree the question (as stated) is a bit strange but I did that to simplify the question. The real world usecase would indeed be tracking something dynamic like a job that processes a number of DB rows and I need to know how many DB rows I processed at 03:14 last night (for example). Thanks for confirming my initial thought that the push gateway is a good option here :)
Answers: Ezariah A
Answers: CC BY-SA 4.0
Answers: 2022-08-09T04:42:23.370Z
Answers: DB rows processed makes sense as a metric. I think you should, at least, explore using Pushgateway for your Jobs. Good luck!
Answers: DazWilkin
Answers: CC BY-SA 4.0
Answers: 2022-08-09T02:20:15.843Z
Answers: 2022-08-09T02:20:15.843Z
Answers: DazWilkin
Body: <p>I want to create a Kubernetes Job object like the example: <a href="https://kubernetes.io/docs/concepts/workloads/controllers/job/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/controllers/job/</a></p>
  <p>Now imagine I want to have the result (in this case the value of Pi: <code>3.14159...</code>) available in Prometheus.</p>
  <p>Is this possible?</p>
  <p>In a more complex example, imagine the output of my pod was JSON:</p>
  <pre><code>{
    &quot;foo&quot;: 200,
    &quot;bar&quot;: 10,
    &quot;baz&quot;: 999
  }
  </code></pre>
  <p>I somehow need to denote that I'm interested in <code>foo</code> and <code>baz</code> (but not <code>bar</code>) &quot;being available&quot; in Prom (note I'm not opinionated on whether the solution is push or pull.</p>
  <h2>Option 1</h2>
  <p>The first option I've thought of is somehow attaching the result <code>3.14159</code> to the job and Prom can scrape it as normal.</p>
  <h2>Option 2</h2>
  <p>Don't run the Pi-generator container at all, instead run something like a Python script that runs the Pi process. Then we can push from Python to Prom.</p>
  <h2>Option 3</h2>
  <p>Some other way?</p>
  
Comments: CC BY-SA 4.0
Comments: 2022-08-09T02:02:09.070Z
Comments: Kudos to you for taking the commenter's feedback on Stack overflow and "moving" the question here.
Comments: DazWilkin
Tags: <kubernetes><prometheus>
Title: Kubernetes Job Metrics in Prometheus
UserDisplayName: Ezariah A
ThreadText: # Kubernetes Job Metrics in Prometheus
  
  (Asked by: username_0 on 2022-08-08)
  
  I want to create a Kubernetes Job object like the example: <https://kubernetes.io/docs/concepts/workloads/controllers/job/>
  
  Now imagine I want to have the result (in this case the value of Pi: `3.14159...`) available in Prometheus.
  
  Is this possible?
  
  In a more complex example, imagine the output of my pod was JSON:
  
  ```
  {
    "foo": 200,
    "bar": 10,
    "baz": 999
  }
  
  ```
  
  I somehow need to denote that I'm interested in `foo` and `baz` (but not `bar`) "being available" in Prom (note I'm not opinionated on whether the solution is push or pull.
  
  ## Option 1
  
  The first option I've thought of is somehow attaching the result `3.14159` to the job and Prom can scrape it as normal.
  
  ## Option 2
  
  Don't run the Pi-generator container at all, instead run something like a Python script that runs the Pi process. Then we can push from Python to Prom.
  
  ## Option 3
  
  Some other way?
  
  **Tags:** kubernetes, prometheus
  
  > **Comments:**
  > **username_1**: Kudos to you for taking the commenter's feedback on Stack overflow and "moving" the question here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-09. Score: 2
  
  I think your need may be an anti-pattern but I'm unsure.
  
  Metrics are used to measure things and Metrics are correlated with time.
  
  In your example, the job's output is a constant and not correlated with a time.
  
  Metrics are often (!) measurements of the health (of the state) of a system rather than the output (product) of a system.
  
  The job's duration, CPU, memory, success|failure etc. are conventional (!) measurements.
  
  While it's entirely reasonable to want to capture time-series data (from Jobs), it may be (!) that a database or some other persistence mechanism would be a better sink for your data.
  
  Answering your question as stated: Jobs are challenging because they are potentially unscrapeable (because they don't live sufficiently long) by Prometheus as part of its 'pull' mechanism.
  
  Batch jobs are a valid use-case for [Pushgateway](https://prometheus.io/docs/practices/pushing/). So, your Jobs could push metrics to Pushgateway to ensure that these are captured.
  
  If the logs produced by your Jobs are persisted beyond the life of the Job, another approach is to derive metrics (by parsing) logs. This approach biases towards aggregation of log data. The examplar is counting HTTP 500s in log entries to determine a failure rate.
  
  > **Comments:**
  > **username_0**: Thanks Dan, I agree the question (as stated) is a bit strange but I did that to simplify the question. The real world usecase would indeed be tracking something dynamic like a job that processes a number of DB rows and I need to know how many DB rows I processed at 03:14 last night (for example). Thanks for confirming my initial thought that the push gateway is a good option here :)  
  > **username_1**: DB rows processed makes sense as a metric. I think you should, at least, explore using Pushgateway for your Jobs. Good luck!  
  
  ---
  
  