title: Container Fails to Start: Insufficient memory for the Java Runtime Environment to continue
body: <p>We have an enterprise application running on Java 8. The deployment environment is built &amp; updated through Bitbucket pipelines. I have a <a href="https://i.stack.imgur.com/0Z3Ga.png" rel="noreferrer">graphic showing the high-level architecture of the environment</a>.  We have two app servers running identical configurations apart from some application specific environment variables.</p>
  <p>It was all working well until a week ago when after a successful pipeline run, the 2 app instances on one of the servers stopped working with the following error:</p>
  <pre><code>There is insufficient memory for the Java Runtime Environment to continue.
  Cannot create GC thread. Out of system resources.
  </code></pre>
  <p>Both the instances are working fine on the other server. In contrast, the containers fail to start on this server.</p>
  <h2>Solutions Tried</h2>
  <p>The error accompanies the following information:</p>
  <p>Possible reasons:
  The system is out of physical RAM or swap space
  The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap.</p>
  <p>Possible solutions:</p>
  <ul>
  <li>Reduce memory load on the system</li>
  <li>Increase physical memory or swap space</li>
  <li>Check if swap backing store is full</li>
  <li>Decrease Java heap size (-Xmx/-Xms)</li>
  <li>Decrease number of Java threads</li>
  <li>Decrease Java thread stack sizes (-Xss)</li>
  <li>Set larger code cache with -XX:ReservedCodeCacheSize=</li>
  </ul>
  <p>We have tried:</p>
  <ol>
  <li>Adding more swap memory. The server has 8GB of RAM while we have tried the swap from 4GB to 9GB.</li>
  <li>Played with the heap sizes Xms &amp; Xmx from 128m to 4096m.</li>
  <li>Increased the RAM on this server to 16GB while the other server that works still does on 8GB.</li>
  </ol>
  <p>Here is how the memory &amp; swap consumption looks like:</p>
  <pre><code>free -mh
                total        used        free      shared  buff/cache   available
  Mem:           15Gi       378Mi        12Gi       1.0Mi       2.9Gi        14Gi
  Swap:           9Gi          0B         9Gi
  </code></pre>
  <p>I have links to several related artifacts.  These include the <a href="https://pastebin.mozilla.org/tE82QzDQ" rel="noreferrer">complete <code>docker logs</code> output</a> and the output of <code>docker info</code> on <a href="https://pastebin.mozilla.org/DdgUDpwj" rel="noreferrer">the failing server</a> and <a href="https://pastebin.mozilla.org/GrzhTFBT" rel="noreferrer">the operational server</a>.</p>
  <p>This is what <code>docker ps -a</code> gets us:</p>
  <pre><code>:~$ docker ps -a
  CONTAINER ID   IMAGE                                                                                  COMMAND                  CREATED        STATUS                    PORTS                                       NAMES
  d29747bf2ad3   :a7608a838625ae945bd0a06fea9451f8bf11ebe4   &quot;catalina.sh run&quot;        10 hours ago   Exited (1) 10 hours ago                                               jbbatch
  0951b6eb5d42   :a7608a838625ae945bd0a06fea9451f8bf11ebe4   &quot;catalina.sh run&quot;        10 hours ago   Exited (1) 10 hours ago                                               jbapp
  </code></pre>
  <p>We are out of ideas right now as we have tried almost all the solutions on stack overflow. What are we missing?</p>