Answers: <p>Regardless if containers are ephemeral or not, there's two things you could consider <em>not normal</em>:</p>
  
  <ul>
  <li>Non-zero exit codes</li>
  <li>Restart count > 0 at the orchestration layer</li>
  </ul>
  
  <p>How you can alert on those metrics will vary wildly between orchestration layers, providers and monitoring tool. I'll keep the discussion about the <em>principles</em>:</p>
  
  <h2>Non-zero exit codes</h2>
  
  <p>This one seems obvious enough. If you have an ephemeral workload, make sure it exits cleanly and you'll be able to monitor the health of your containers. Even if your ephemeral workload last 30 seconds and you launch thousand of them (small short-lived workers style), as long as everything exits cleanly, there's nothing to worry about.</p>
  
  <h2>Restart count at the orchestration layer</h2>
  
  <p>Most orchestration layers will expose a restart count. For Kubernetes, that's literally the 'Restart Count' metric when describing a pod. Those restarts are the orchestration layer restarting a container because it wasn't supposed to be in a 'stopped' state, but it was. Ephemeral workloads won't have a restart count, because of their very nature (or if they do, the "final" shutdown of the container won't cause a restart when the workload is done).</p>
  
  <p>Whether you want to alert as soon as the restart count crosses a certain threshold or if you want to be more clever about it and calculate a rate (roughly: restart count / pod/service duration) is up to you.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-05-22T21:25:32.157Z
Answers: Do you know if ECS surfaces the restart count metric?
Answers: MrDuk
Answers: CC BY-SA 4.0
Answers: 2019-05-23T00:34:13.693Z
Answers: Not through CloudWatch (sadly), but yes, this information is available even if it requires a bit plumbing.
  
  Both the 'Service' and the 'Task' abstractions in ECS give you insightful information.
  Service: events and failures
  Task: healthStatus and failures.
  
  This page https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch_event_stream.html gives you more information on how to stream ECS events so you can make use of them in an automated way.
  
  For what it's worth, I've personally used Prometheus ECS Exporter in the past and I was alerting with the metrics it was exposing.
Answers: Alexandre
Answers: CC BY-SA 4.0
Answers: 2019-05-22T15:10:53.677Z
Answers: 2019-05-22T15:10:53.677Z
Answers: Alexandre
Body: <p>Are there any proposed best practices for how to monitor for thrashing containers? We're in a situation where we have containers that might try to come up and will crash -> restart a few dozen times without anyone noticing. However, it's "normal" for containers to be so ephemeral, so tracking this sort of anomaly seems difficult (what if your workload really is just 30 seconds?)</p>
  
  <p>I'm just wondering if there are any "good enough" alerting practices that anyone else has come up with for tracking unhealthy restarts vs healthy restarts?</p>
  
Tags: <containers><monitoring>
Title: How do you monitor and alert on thrashing containers?
UserDisplayName: MrDuk
ThreadText: # How do you monitor and alert on thrashing containers?
  
  (Asked by: username_0 on 2019-05-21)
  
  Are there any proposed best practices for how to monitor for thrashing containers? We're in a situation where we have containers that might try to come up and will crash -> restart a few dozen times without anyone noticing. However, it's "normal" for containers to be so ephemeral, so tracking this sort of anomaly seems difficult (what if your workload really is just 30 seconds?)
  
  I'm just wondering if there are any "good enough" alerting practices that anyone else has come up with for tracking unhealthy restarts vs healthy restarts?
  
  **Tags:** containers, monitoring
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-22. Score: 3 | Accepted answer
  
  Regardless if containers are ephemeral or not, there's two things you could consider *not normal*:
  
  * Non-zero exit codes
  * Restart count > 0 at the orchestration layer
  
  How you can alert on those metrics will vary wildly between orchestration layers, providers and monitoring tool. I'll keep the discussion about the *principles*:
  
  ## Non-zero exit codes
  
  This one seems obvious enough. If you have an ephemeral workload, make sure it exits cleanly and you'll be able to monitor the health of your containers. Even if your ephemeral workload last 30 seconds and you launch thousand of them (small short-lived workers style), as long as everything exits cleanly, there's nothing to worry about.
  
  ## Restart count at the orchestration layer
  
  Most orchestration layers will expose a restart count. For Kubernetes, that's literally the 'Restart Count' metric when describing a pod. Those restarts are the orchestration layer restarting a container because it wasn't supposed to be in a 'stopped' state, but it was. Ephemeral workloads won't have a restart count, because of their very nature (or if they do, the "final" shutdown of the container won't cause a restart when the workload is done).
  
  Whether you want to alert as soon as the restart count crosses a certain threshold or if you want to be more clever about it and calculate a rate (roughly: restart count / pod/service duration) is up to you.
  
  > **Comments:**
  > **username_0**: Do you know if ECS surfaces the restart count metric?  
  > **username_1**: Not through CloudWatch (sadly), but yes, this information is available even if it requires a bit plumbing.  Both the 'Service' and the 'Task' abstractions in ECS give you insightful information. Service: events and failures Task: healthStatus and failures.  This page https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch_event_stream.html gives you more information on how to stream ECS events so you can make use of them in an automated way.  For what it's worth, I've personally used Prometheus ECS Exporter in the past and I was alerting with the metrics it was exposing.  
  
  ---
  
  