Answers: <p>From the get-go, my educated guess would be memory leak your workloads. It seems that his happens pretty regular - every 12 hours, do you have some scheduled/cron job?</p>
  <p>Alternative problem could be overcommitted resources.</p>
  <p>First of all check <code>resources.limits</code> on workloads you are running on the EKS cluster. Make sure <code>resources.limits.memory</code> is set on every single container. You can play around with <a href="https://kubernetes.io/docs/concepts/policy/limit-range/" rel="nofollow noreferrer">limit ranges</a> to set defaults for a namespace; this way you &quot;protect&quot; your nodes to some extent (alternative is to use <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="nofollow noreferrer">resource quotas</a>.</p>
  <p>I see you are using Datadog, you can leverage Kubernetes metrics to find spiking workload - <code>kubernetes.memory.working_set</code> (<a href="https://docs.datadoghq.com/containers/kubernetes/data_collected/" rel="nofollow noreferrer">reference</a>):</p>
  <blockquote>
  <p>Current working set in bytes - this is what the OOM killer is watching for</p>
  </blockquote>
  <p>The description of metric you are currently using (<a href="https://docs.datadoghq.com/metrics/guide/calculating-the-system-mem-used-metric/" rel="nofollow noreferrer">reference</a>):</p>
  <blockquote>
  <p>The <code>system.mem.usable</code> metric represents free memory plus cached memory plus buffers (on Linux, it reflects “MemAvailable” attribute from <code>/proc/meminfo</code> whenever possible).</p>
  </blockquote>
  <p>On a &quot;cluster&quot; side of things you may enable <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/" rel="nofollow noreferrer">&quot;resource reservation&quot;</a>, which may help with keeping system daemons (like <code>kubelet</code>) alive. Also read docs on node <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#hard-eviction-thresholds" rel="nofollow noreferrer">pressure evictions and thresholds</a>.</p>
  
Answers: CC BY-SA 4.0
Answers: 2023-06-24T13:10:19.317Z
Answers: 2023-06-24T13:10:19.317Z
Answers: Kyslik
Body: <p>I was able to correlate the &quot;usable memory&quot; going down with events in kubernetes:</p>
  <pre><code>Conditions:
    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----             ------  -----------------                 ------------------                ------                       -------
    MemoryPressure   False   Tue, 06 Jun 2023 19:31:34 +0000   Thu, 11 May 2023 15:24:32 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  </code></pre>
  <p>The time of the <code>LastTransitionTime</code> coincides with the lowest memory and the transition to a lot of available memory.</p>
  <p>Also, I think that during the times of low available memory I get a lot of OOM errors and processes are getting killed with exit code 137.</p>
  <p>Is there something misconfigured in this Kubernetes cluster? I think the available memory should not ever go that low and it should start evicting pods earlier. Also I don't understand why the whole memory becomes available - this means that we are evicting ALL pods?</p>
  <p><a href="https://i.sstatic.net/MamJK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MamJK.png" alt="memory available goes lower and lower then jumps up" /></a></p>
  
Tags: <kubernetes><eks>
Title: EKS: Usable Memory goes lower and lower on the node to 5% then jumps up to 80% available
UserDisplayName: Costi
ThreadText: # EKS: Usable Memory goes lower and lower on the node to 5% then jumps up to 80% available
  
  (Asked by: username_0 on 2023-06-06)
  
  I was able to correlate the "usable memory" going down with events in kubernetes:
  
  ```
  Conditions:
    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----             ------  -----------------                 ------------------                ------                       -------
    MemoryPressure   False   Tue, 06 Jun 2023 19:31:34 +0000   Thu, 11 May 2023 15:24:32 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  
  ```
  
  The time of the `LastTransitionTime` coincides with the lowest memory and the transition to a lot of available memory.
  
  Also, I think that during the times of low available memory I get a lot of OOM errors and processes are getting killed with exit code 137.
  
  Is there something misconfigured in this Kubernetes cluster? I think the available memory should not ever go that low and it should start evicting pods earlier. Also I don't understand why the whole memory becomes available - this means that we are evicting ALL pods?
  
  [![memory available goes lower and lower then jumps up](https://i.sstatic.net/MamJK.png)](https://i.sstatic.net/MamJK.png)
  
  **Tags:** kubernetes, eks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-24. Score: 0
  
  From the get-go, my educated guess would be memory leak your workloads. It seems that his happens pretty regular - every 12 hours, do you have some scheduled/cron job?
  
  Alternative problem could be overcommitted resources.
  
  First of all check `resources.limits` on workloads you are running on the EKS cluster. Make sure `resources.limits.memory` is set on every single container. You can play around with [limit ranges](https://kubernetes.io/docs/concepts/policy/limit-range/) to set defaults for a namespace; this way you "protect" your nodes to some extent (alternative is to use [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/).
  
  I see you are using Datadog, you can leverage Kubernetes metrics to find spiking workload - `kubernetes.memory.working_set` ([reference](https://docs.datadoghq.com/containers/kubernetes/data_collected/)):
  
  > Current working set in bytes - this is what the OOM killer is watching for
  
  The description of metric you are currently using ([reference](https://docs.datadoghq.com/metrics/guide/calculating-the-system-mem-used-metric/)):
  
  > The `system.mem.usable` metric represents free memory plus cached memory plus buffers (on Linux, it reflects “MemAvailable” attribute from `/proc/meminfo` whenever possible).
  
  On a "cluster" side of things you may enable ["resource reservation"](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/), which may help with keeping system daemons (like `kubelet`) alive. Also read docs on node [pressure evictions and thresholds](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#hard-eviction-thresholds).
  
  
  ---
  
  