<p dir="auto"><strong>Apache Airflow version</strong>: 2.0.2</p>
<p dir="auto"><strong>Kubernetes version</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;19&quot;, GitVersion:&quot;v1.19.3&quot;, GitCommit:&quot;1e11e4a2108024935ecfcb2912226cedeafd99df&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2020-10-14T12:50:19Z&quot;, GoVersion:&quot;go1.15.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;18+&quot;, GitVersion:&quot;v1.18.17-gke.1901&quot;, GitCommit:&quot;b5bc948aea9982cd8b1e89df8d50e30ffabdd368&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-05-27T19:56:12Z&quot;, GoVersion:&quot;go1.13.15b4&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.17-gke.1901", GitCommit:"b5bc948aea9982cd8b1e89df8d50e30ffabdd368", GitTreeState:"clean", BuildDate:"2021-05-27T19:56:12Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>: Kubernetes setup where the schedule component ran its own pod.</p>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Scheduler memory crept up for no apparent reason given a deployment with two simple DAGs.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/5952735/124017862-111a6480-d99c-11eb-8f0d-b92ca55bffd1.png"><img src="https://user-images.githubusercontent.com/5952735/124017862-111a6480-d99c-11eb-8f0d-b92ca55bffd1.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I expect the scheduler memory usage to not build up over time and should remain relatively flat throughout the scheduler process lifetime.</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<p dir="auto">Start a scheduler process in a pod and watch the memory usage over time.</p>
<p dir="auto">These are the scheduler configurations</p>
<ul dir="auto">
<li>1 CPU and 3.75 GB memory</li>
<li>CeleryExecutor</li>
<li>2 parsing processes</li>
<li>2 simple DAGs</li>
</ul>
<p dir="auto">Hey <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ashb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ashb">@ashb</a> -  following your request of observing the self-hosted behavior.</p>
<p dir="auto">I believe the current scale-in settings are slightly too aggressive - especially when the traffic is low (weekends).</p>
<p dir="auto">I pushed a number of builds here (and <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/turbaszek/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/turbaszek">@turbaszek</a> as well):</p>
<p dir="auto"><a href="https://github.com/apache/airflow/actions/workflows/build-images-workflow-run.yml">https://github.com/apache/airflow/actions/workflows/build-images-workflow-run.yml</a></p>
<p dir="auto">And quite a number a lot of them failed without logs indicating the scale-in event happened. some of them with 'git' failed, some of them with explicitly <code class="notranslate">lost communication</code></p>
<p dir="auto">Some examples here:</p>
<ul dir="auto">
<li><a href="https://github.com/apache/airflow/actions/runs/651685483">https://github.com/apache/airflow/actions/runs/651685483</a></li>
<li><a href="https://github.com/apache/airflow/actions/runs/651569791">https://github.com/apache/airflow/actions/runs/651569791</a></li>
<li><a href="https://github.com/apache/airflow/actions/runs/651571502">https://github.com/apache/airflow/actions/runs/651571502</a></li>
<li><a href="https://github.com/apache/airflow/actions/runs/651550722">https://github.com/apache/airflow/actions/runs/651550722</a></li>
<li><a href="https://github.com/apache/airflow/actions/runs/651642721">https://github.com/apache/airflow/actions/runs/651642721</a></li>
<li><a href="https://github.com/apache/airflow/actions/runs/651688017">https://github.com/apache/airflow/actions/runs/651688017</a></li>
</ul>
<p dir="auto">(maybe some of those were cancelled as duplicates - but at most 1 or 2)</p>
<p dir="auto">At the same time a number of those jobs succeeded, so I think the scale-in events are the ones to blame.</p>
<p dir="auto">The previous setting was much more stable (but more costly as well) - however I think I will merge the <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="818258641" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/14531" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/14531/hovercard" href="https://github.com/apache/airflow/pull/14531">#14531</a> which should sigificantly decrease the time needed from the runners so hopefully we will be able to tune up the scale-in settings so that they are more stable.</p>