title: Can you serve Kubernetes liveness endpoint with dedicated connections, that are available when regular connection pool gets exhausted?
body: <p>When our PODs gets exhausted and could need some scaling up, Kubernetes confuses this exhaustion with death and restarts our POD:s. This has, of course, the opposite effect, the remaining POD:s gets even more load... So here comes my question, can you serve Kubernetes and LB liveness and readiness endpoints with dedicated, non-exhausted, connections?</p>
  <p>We have an older system running in Kubernetes, one Apache httpd and one tomcat bundled in each POD. Load balancing is done by Kubernetes between different POD:s, not in httpd. Httpd is running mpm_event+mod_jk and there is an AJP 1.3 connection to the Tomcat. Httpd is also serving some static resources from disc without the Tomcat. When something fails, we quickly run out of AJP threads and HTTPD workers.</p>
  <p>Basically what we see is this:</p>
  <ol>
  <li>The application fails to connect to some resource. Some network, Memcached, DB or other service starts to time out. Waiting on timeouts causes threads to be very long-lived and we run out of them quickly.</li>
  <li>Readiness/Liveness probs do not respond in time, Kubernetes restarts the POD (or, after we removed the liveness probe, the LB that uses readiness removes them from load balancing, having basically the same effect).</li>
  <li>Root cause problem is solved (somehow), but now there are too few (on non) POD:s left in Load balancing. When a POD gets back, it's hit by all the traffic, gets exhausted, and are removed from LB since it's too slow on the readiness probe again.</li>
  <li>We now find it very difficult to get out of this state... (So far it happened twice, and we basically had to cut off all traffic on Cloudflare WAF until enough POD:s were restarted/in Loadbalancing...)</li>
  </ol>
  <p><strong>My idea of a solution:</strong></p>
  <p>I think I can open a prioritised fastlane from httpd-&gt;tomcat for the liveness and readiness endpoints, see below. But, can I somehow dedicate workers in httpd (mpm_event) to these endpoints? Else, when I run out of httpd workers, my fast lane will not offer any help I guess. Or any other ideas about how to ensure that we always can serve liveness/readiness as long as the tomcat is alive, even when it is exhausted?</p>
  <p>This is my current httpd worker setup:</p>
  <pre class="lang-xml prettyprint-override"><code>&lt;IfModule mpm_event_module&gt;
      StartServers             3
      ServerLimit             36
      MinSpareThreads         75
      MaxSpareThreads        250
      ThreadsPerChild         25
      MaxRequestWorkers      900
      MaxConnectionsPerChild   0
  &lt;/IfModule&gt;
  </code></pre>
  <p>Maybe it takes a worker just to analyze the request and figure out the URI... :-/ Or can I somehow dedicate a specific pool of workers to liveness and readiness???</p>
  <p><strong>My httpd-&gt;tomcat fastlane:</strong></p>
  <p>I was playing around with a second AJP connection to the tomcat, dedicated to the readiness and liveness endpoints. At a glance, it seems to work.</p>
  <p>In server.xml I added a connector on port 8008:</p>
  <pre class="lang-xml prettyprint-override"><code>    &lt;Connector
          port=&quot;8009&quot;
          protocol=&quot;AJP/1.3&quot;
          redirectPort=&quot;8443&quot;
          connectionTimeout=&quot;60000&quot;
          minSpareThreads=&quot;2&quot;
          maxThreads=&quot;20&quot;
          acceptorThreadCount=&quot;2&quot;
          URIEncoding=&quot;UTF-8&quot;
          address=&quot;127.0.0.1&quot;
          secretRequired=&quot;false&quot; /&gt;
  
      &lt;!--
        This is the prioritized connector used for health checks.
      --&gt;
      &lt;Connector
          port=&quot;8008&quot;
          protocol=&quot;AJP/1.3&quot;
          redirectPort=&quot;8443&quot;
          connectionTimeout=&quot;-1&quot;
          keepAliveTimeout=&quot;-1&quot;
          acceptorThreadPriority=&quot;6&quot;
          minSpareThreads=&quot;2&quot;
          maxThreads=&quot;5&quot;
          acceptorThreadCount=&quot;1&quot;
          URIEncoding=&quot;UTF-8&quot;
          address=&quot;127.0.0.1&quot;
          secretRequired=&quot;false&quot; /&gt;
  </code></pre>
  <p>In my <code>workers.properties</code> (the JkWorkersFile) I added the new connection and named it <code>ajp13prio</code>:</p>
  <pre class="lang-yaml prettyprint-override"><code>worker.list=ajp13,ajp13prio
  worker.ajp13.type=ajp13
  worker.ajp13.port=8009
  worker.ajp13.host=127.0.0.1
  worker.ajp13.lbfactor=1
  worker.ajp13prio.type=ajp13
  worker.ajp13prio.port=8008
  worker.ajp13prio.host=127.0.0.1
  worker.ajp13prio.lbfactor=1
  </code></pre>
  <p>In my httpd conf I configured the probes to use the new connector:</p>
  <pre class="lang-xml prettyprint-override"><code>&lt;VirtualHost *:80&gt;
  ...
      # health checks (readiness and liveness probes) are prioritized
      JkMount /api/v2/health/* ajp13prio
  
      # All requests go to worker1 by default
      JkMount /* ajp13
  ...
  &lt;/VirtualHost&gt;
  </code></pre>