<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.2.1</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">When I try to backfill using <code class="notranslate">airflow dags backfill ...</code>, the 'Run' shows running but the tasks are stuck in scheduled state forever.</p>
<ol dir="auto">
<li>Code for DAG to reproduce the problem</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import time
from datetime import timedelta

import pendulum
from airflow import DAG
from airflow.decorators import task
from airflow.models.dag import dag
from airflow.operators.bash import BashOperator
from airflow.operators.dummy import DummyOperator
from airflow.operators.python import PythonOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['airflow@example.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 2,
    'retry_delay': timedelta(minutes=5),
}


def get_execution_date(**kwargs):
    ds = kwargs['ds']
    print(ds)

with DAG(
        'test_dag_2',
        default_args=default_args,
        description='Testing dag',
        start_date=pendulum.datetime(2022, 4, 2, tz='UTC'),
        schedule_interval=&quot;@hourly&quot;, max_active_runs=5, concurrency=10,
) as dag:
    t1 = BashOperator(
        task_id='task_1',
        depends_on_past=False,
        bash_command='sleep 5'
    )

    t2 = PythonOperator(
        task_id='get_execution_date',
        python_callable=get_execution_date
    )

    t1 &gt;&gt; t2"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">time</span>
<span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">timedelta</span>

<span class="pl-k">import</span> <span class="pl-s1">pendulum</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span> <span class="pl-k">import</span> <span class="pl-v">DAG</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">decorators</span> <span class="pl-k">import</span> <span class="pl-s1">task</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">models</span>.<span class="pl-s1">dag</span> <span class="pl-k">import</span> <span class="pl-s1">dag</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">bash</span> <span class="pl-k">import</span> <span class="pl-v">BashOperator</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">dummy</span> <span class="pl-k">import</span> <span class="pl-v">DummyOperator</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">python</span> <span class="pl-k">import</span> <span class="pl-v">PythonOperator</span>

<span class="pl-s1">default_args</span> <span class="pl-c1">=</span> {
    <span class="pl-s">'owner'</span>: <span class="pl-s">'airflow'</span>,
    <span class="pl-s">'depends_on_past'</span>: <span class="pl-c1">False</span>,
    <span class="pl-s">'email'</span>: [<span class="pl-s">'airflow@example.com'</span>],
    <span class="pl-s">'email_on_failure'</span>: <span class="pl-c1">True</span>,
    <span class="pl-s">'email_on_retry'</span>: <span class="pl-c1">False</span>,
    <span class="pl-s">'retries'</span>: <span class="pl-c1">2</span>,
    <span class="pl-s">'retry_delay'</span>: <span class="pl-en">timedelta</span>(<span class="pl-s1">minutes</span><span class="pl-c1">=</span><span class="pl-c1">5</span>),
}


<span class="pl-k">def</span> <span class="pl-en">get_execution_date</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-s1">kwargs</span>[<span class="pl-s">'ds'</span>]
    <span class="pl-en">print</span>(<span class="pl-s1">ds</span>)

<span class="pl-k">with</span> <span class="pl-v">DAG</span>(
        <span class="pl-s">'test_dag_2'</span>,
        <span class="pl-s1">default_args</span><span class="pl-c1">=</span><span class="pl-s1">default_args</span>,
        <span class="pl-s1">description</span><span class="pl-c1">=</span><span class="pl-s">'Testing dag'</span>,
        <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">pendulum</span>.<span class="pl-en">datetime</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">2</span>, <span class="pl-s1">tz</span><span class="pl-c1">=</span><span class="pl-s">'UTC'</span>),
        <span class="pl-s1">schedule_interval</span><span class="pl-c1">=</span><span class="pl-s">"@hourly"</span>, <span class="pl-s1">max_active_runs</span><span class="pl-c1">=</span><span class="pl-c1">5</span>, <span class="pl-s1">concurrency</span><span class="pl-c1">=</span><span class="pl-c1">10</span>,
) <span class="pl-k">as</span> <span class="pl-s1">dag</span>:
    <span class="pl-s1">t1</span> <span class="pl-c1">=</span> <span class="pl-v">BashOperator</span>(
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'task_1'</span>,
        <span class="pl-s1">depends_on_past</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
        <span class="pl-s1">bash_command</span><span class="pl-c1">=</span><span class="pl-s">'sleep 5'</span>
    )

    <span class="pl-s1">t2</span> <span class="pl-c1">=</span> <span class="pl-v">PythonOperator</span>(
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'get_execution_date'</span>,
        <span class="pl-s1">python_callable</span><span class="pl-c1">=</span><span class="pl-s1">get_execution_date</span>
    )

    <span class="pl-s1">t1</span> <span class="pl-c1">&gt;&gt;</span> <span class="pl-s1">t2</span></pre></div>
<ol start="2" dir="auto">
<li>Airflow backfill CLI</li>
</ol>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ airflow dags backfill --subdir /opt/airflow/dags/repo --reset-dagruns -s &quot;2022-03-01 01:00:00&quot; -e &quot;2022-03-02 01:00:00&quot; test_dag_2"><pre class="notranslate">$ airflow dags backfill --subdir /opt/airflow/dags/repo --reset-dagruns -s <span class="pl-s"><span class="pl-pds">"</span>2022-03-01 01:00:00<span class="pl-pds">"</span></span> -e <span class="pl-s"><span class="pl-pds">"</span>2022-03-02 01:00:00<span class="pl-pds">"</span></span> test_dag_2</pre></div>
<ol start="3" dir="auto">
<li>Result</li>
</ol>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/36870121/168201159-fcdf7640-2bf9-428f-849e-25f18dd2b504.png"><img width="472" alt="image" src="https://user-images.githubusercontent.com/36870121/168201159-fcdf7640-2bf9-428f-849e-25f18dd2b504.png" style="max-width: 100%;"></a></p>
<ol start="4" dir="auto">
<li>Discussion<br>
The 'Run' shows running but the tasks are stuck in scheduled state forever.<br>
Here are some variants what I have checked. (but still have this issue.)</li>
</ol>
<ul dir="auto">
<li>max_active_runs, concurrency</li>
<li>-x option for backfill command</li>
<li>--subdir option for backfill command</li>
<li>--reset-dagruns for backfill command</li>
<li>schedule_interval to <code class="notranslate">None</code> and <code class="notranslate">@once</code></li>
<li>Restart the Airflow scheduler pods.</li>
<li><strong>Upgrade Airflow version to 2.3.0.</strong></li>
</ul>
<ol start="5" dir="auto">
<li>Related issue<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="781406229" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/13542" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/13542/hovercard" href="https://github.com/apache/airflow/issues/13542">#13542</a></li>
</ol>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">The backfill jobs should successfully done from the past.</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">CentOS Linux release 7.9.2009 (Core)</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto">apache-airflow-providers-amazon          3.3.0<br>
apache-airflow-providers-celery          2.1.0<br>
apache-airflow-providers-cncf-kubernetes 3.0.2<br>
apache-airflow-providers-docker          2.6.0<br>
apache-airflow-providers-elasticsearch   3.0.3<br>
apache-airflow-providers-ftp             2.1.2<br>
apache-airflow-providers-grpc            2.0.4<br>
apache-airflow-providers-hashicorp       2.2.0<br>
apache-airflow-providers-http            2.0.2<br>
apache-airflow-providers-imap            2.2.3<br>
apache-airflow-providers-postgres        4.1.0<br>
apache-airflow-providers-redis           2.0.4<br>
apache-airflow-providers-sendgrid        2.0.4<br>
apache-airflow-providers-sftp            2.6.0<br>
apache-airflow-providers-slack           4.2.3<br>
apache-airflow-providers-sqlite          2.1.3<br>
apache-airflow-providers-ssh             2.4.3</p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Official Apache Airflow Helm Chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto">Here are some logs from scheduler pod, found kubernetes_executor try to delete worker pod.</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[2022-05-13 02:48:08,417] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type MODIFIED
[2022-05-13 02:48:08,417] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,423] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of type DELETED
[2022-05-13 02:48:08,423] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,497] {backfill_job.py:397} INFO - [backfill progress] | finished run 0 of 25 | tasks waiting: 5 | succeeded: 0 | running: 5| failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2022-05-13 02:48:13,390] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,391] {kubernetes_executor.py:375} INFO - Attempting to finish pod; pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a; state: None; annotations: {'dag_id': 'test_dag_2', 'task_id': 'task_1', 'execution_date': None, 'run_id': 'backfill__2022-03-03T02:00:00+00:00', 'try_number': '1'}
[2022-05-13 02:48:13,392] {kubernetes_executor.py:576} INFO - Changing state of (TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1), None, 'testdag2task1.34ae944f04ee4a18a65f559163ca0d1a', 'jutopia-chlee-test-chlee-backfill-test', '3559062271') to None
[2022-05-13 02:48:13,396] {kubernetes_executor.py:661} INFO - Deleted pod: TaskInstanceKey(dag_id='test_dag_2', task_id='task_1', run_id='backfill__2022-03-03T02:00:00+00:00', try_number=1) in namespace jutopia-chlee-test-chlee-backfill-test"><pre class="notranslate">[2022-05-13 02:48:08,417] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of <span class="pl-c1">type</span> MODIFIED
[2022-05-13 02:48:08,417] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,423] {kubernetes_executor.py:147} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a had an event of <span class="pl-c1">type</span> DELETED
[2022-05-13 02:48:08,423] {kubernetes_executor.py:206} INFO - Event: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a Succeeded
[2022-05-13 02:48:08,497] {backfill_job.py:397} INFO - [backfill progress] <span class="pl-k">|</span> finished run 0 of 25 <span class="pl-k">|</span> tasks waiting: 5 <span class="pl-k">|</span> succeeded: 0 <span class="pl-k">|</span> running: 5<span class="pl-k">|</span> failed: 0 <span class="pl-k">|</span> skipped: 0 <span class="pl-k">|</span> deadlocked: 0 <span class="pl-k">|</span> not ready: 5
[2022-05-13 02:48:13,390] {kubernetes_executor.py:375} INFO - Attempting to finish pod<span class="pl-k">;</span> pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a<span class="pl-k">;</span> state: None<span class="pl-k">;</span> annotations: {<span class="pl-s"><span class="pl-pds">'</span>dag_id<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>test_dag_2<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>task_id<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>task_1<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>execution_date<span class="pl-pds">'</span></span>: None, <span class="pl-s"><span class="pl-pds">'</span>run_id<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>backfill__2022-03-03T02:00:00+00:00<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>try_number<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>1<span class="pl-pds">'</span></span>}
[2022-05-13 02:48:13,391] {kubernetes_executor.py:375} INFO - Attempting to finish pod<span class="pl-k">;</span> pod_id: testdag2task1.34ae944f04ee4a18a65f559163ca0d1a<span class="pl-k">;</span> state: None<span class="pl-k">;</span> annotations: {<span class="pl-s"><span class="pl-pds">'</span>dag_id<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>test_dag_2<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>task_id<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>task_1<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>execution_date<span class="pl-pds">'</span></span>: None, <span class="pl-s"><span class="pl-pds">'</span>run_id<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>backfill__2022-03-03T02:00:00+00:00<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>try_number<span class="pl-pds">'</span></span>: <span class="pl-s"><span class="pl-pds">'</span>1<span class="pl-pds">'</span></span>}
[2022-05-13 02:48:13,392] {kubernetes_executor.py:576} INFO - Changing state of (TaskInstanceKey(dag_id=<span class="pl-s"><span class="pl-pds">'</span>test_dag_2<span class="pl-pds">'</span></span>, task_id=<span class="pl-s"><span class="pl-pds">'</span>task_1<span class="pl-pds">'</span></span>, run_id=<span class="pl-s"><span class="pl-pds">'</span>backfill__2022-03-03T02:00:00+00:00<span class="pl-pds">'</span></span>, try_number=1), None, <span class="pl-s"><span class="pl-pds">'</span>testdag2task1.34ae944f04ee4a18a65f559163ca0d1a<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>jutopia-chlee-test-chlee-backfill-test<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>3559062271<span class="pl-pds">'</span></span>) to None
[2022-05-13 02:48:13,396] {kubernetes_executor.py:661} INFO - Deleted pod: TaskInstanceKey(dag_id=<span class="pl-s"><span class="pl-pds">'</span>test_dag_2<span class="pl-pds">'</span></span>, task_id=<span class="pl-s"><span class="pl-pds">'</span>task_1<span class="pl-pds">'</span></span>, run_id=<span class="pl-s"><span class="pl-pds">'</span>backfill__2022-03-03T02:00:00+00:00<span class="pl-pds">'</span></span>, try_number=1) <span class="pl-k">in</span> namespace jutopia-chlee-test-chlee-backfill-test</pre></div>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">Other Airflow 2 version (please specify below)</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">I am using airlfow 2.3.2 on EKS 1.22, the airflow service on EKS was launched by making minor modifications in official helm chart regarding replicas and resources. It is observed that the memory utilization by scheduler pod keeps on increasing as age of scheduler pod increases. Also this is observed when all dags are paused and nothing is running on airlfow. Different versions(config changes) of official helm chat was used to spin different airflow services in the EKS cluster where this issue was observed.</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">This shouldn't have happened, the memory should have remain almost same. The increase in memory made the replicas to increase over few days as replication was setup based on memory utilisation even though there was no changes in the env itself and it was serving any traffic.</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">Should be able to reproduce by using official airflow helm-chart of 2.3.2 on AWS EKS 1.22</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">PRETTY_NAME=“Debian GNU/Linux 11 (bullseye)” NAME=“Debian GNU/Linux” VERSION_ID=“11” VERSION=“11 (bullseye)” VERSION_CODENAME=bullseye ID=debian</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Official Apache Airflow Helm Chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>