<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">Other Airflow 2 version (please specify below)</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">Hello ,</p>
<p dir="auto">We were trying to execute airflow CLI commands for "airflow tasks" in MWAA. While testing these commands , we observed that below "airflow tasks" cli command are parsing parsing all DAGs file then execute provided airflow tasks command.</p>
<p dir="auto"><strong>Commands</strong> :<br>
airflow tasks clear<br>
airflow tasks failed-deps<br>
airflow tasks run<br>
airflow tasks test</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">We are expecting that when to execute "airflow tasks" cli command, it should only parse DAG name and task name provided in the command and not parse DAG lists followed by task list.</p>
<h3 dir="auto">How to reproduce</h3>
<ol dir="auto">
<li>We created web logic token to access web server</li>
<li>After that, we used python script to run "airflow tasks" cli commands.</li>
</ol>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Amazon Linux</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto">Airflow version 2.2.2</p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Amazon (AWS) MWAA</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<p dir="auto">Dear Airflow Maintainers,</p>
<p dir="auto">Before I tell you about my issue, let me describe my environment:</p>
<h1 dir="auto">Environment</h1>
<p dir="auto">Centos 6<br>
CeleryExecutor (redis)<br>
Two workers, 1 scheduler, 1 webserver<br>
MySql</p>
<ul dir="auto">
<li>Version of Airflow 1.6.2.10, 1.7.0rc1, HEAD</li>
<li>Example code to reproduce the bug (as a code snippet in markdown)</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.operators import BashOperator, SubDagOperator



dag_name = 'test_18'
test_pool = 'test_pool'
email_to = 'your@email.com'

now = datetime.now()
start_time = datetime(now.year, now.month, now.day, 12, 0, 0)
start_time = start_time + timedelta(days=-1)

default_args = {
    'owner': 'Test',
    'depends_on_past': True,
    'start_date': start_time,
    'email': [email_to],
    'email_on_failure': True,
    'email_on_retry': True,
    'wait_for_downstream': False,
}

# Create the dag object
dag = DAG(dag_name,
          default_args=default_args,
          schedule_interval='0 * * * *')


def get_subdag(dag, sd_id, pool=None):
    subdag = DAG(
        dag_id='{parent_dag}.{sd_id}'.format(
            parent_dag=dag.dag_id,
            sd_id=sd_id),
        params=dag.params,
        default_args=dag.default_args,
        template_searchpath=dag.template_searchpath,
        user_defined_macros=dag.user_defined_macros,
    )

    t1 = BashOperator(
        task_id='test_task',
        bash_command='echo &quot;hello&quot; &amp;&amp; sleep 10',
        dag=subdag,
        pool=pool
    )

    t2 = BashOperator(
        task_id='test_task_2_with_exit_1',
        bash_command='echo &quot;hello&quot; &amp;&amp; sleep 10 &amp;&amp; exit 1',
        dag=subdag,
        pool=pool
    )

    t2.set_upstream(t1)

    sdo = SubDagOperator(
        task_id=sd_id,
        subdag=subdag,
        retries=2,
        retry_delay=timedelta(seconds=30),
        dag=dag,
        depends_on_past=True,
    )

    return sdo

sd1 = get_subdag(dag, sd_id='test_subdag')
sd2 = get_subdag(dag, sd_id='test_subdag_with_pool', pool=test_pool)"><pre class="notranslate"><code class="notranslate">from datetime import datetime, timedelta
from airflow.models import DAG
from airflow.operators import BashOperator, SubDagOperator



dag_name = 'test_18'
test_pool = 'test_pool'
email_to = 'your@email.com'

now = datetime.now()
start_time = datetime(now.year, now.month, now.day, 12, 0, 0)
start_time = start_time + timedelta(days=-1)

default_args = {
    'owner': 'Test',
    'depends_on_past': True,
    'start_date': start_time,
    'email': [email_to],
    'email_on_failure': True,
    'email_on_retry': True,
    'wait_for_downstream': False,
}

# Create the dag object
dag = DAG(dag_name,
          default_args=default_args,
          schedule_interval='0 * * * *')


def get_subdag(dag, sd_id, pool=None):
    subdag = DAG(
        dag_id='{parent_dag}.{sd_id}'.format(
            parent_dag=dag.dag_id,
            sd_id=sd_id),
        params=dag.params,
        default_args=dag.default_args,
        template_searchpath=dag.template_searchpath,
        user_defined_macros=dag.user_defined_macros,
    )

    t1 = BashOperator(
        task_id='test_task',
        bash_command='echo "hello" &amp;&amp; sleep 10',
        dag=subdag,
        pool=pool
    )

    t2 = BashOperator(
        task_id='test_task_2_with_exit_1',
        bash_command='echo "hello" &amp;&amp; sleep 10 &amp;&amp; exit 1',
        dag=subdag,
        pool=pool
    )

    t2.set_upstream(t1)

    sdo = SubDagOperator(
        task_id=sd_id,
        subdag=subdag,
        retries=2,
        retry_delay=timedelta(seconds=30),
        dag=dag,
        depends_on_past=True,
    )

    return sdo

sd1 = get_subdag(dag, sd_id='test_subdag')
sd2 = get_subdag(dag, sd_id='test_subdag_with_pool', pool=test_pool)
</code></pre></div>
<ul dir="auto">
<li>Screen shots of your DAG's graph and tree views:</li>
</ul>
<ol dir="auto">
<li>sub-task of non pooled subdag fails<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/335167/14066258/b40e237c-f3f9-11e5-83e4-416931a92d92.png"><img width="1440" alt="af_issue_1" src="https://cloud.githubusercontent.com/assets/335167/14066258/b40e237c-f3f9-11e5-83e4-416931a92d92.png" style="max-width: 100%;"></a></li>
<li>subdag enters retry state<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/335167/14066262/c062e8b0-f3f9-11e5-80c9-ef1b09c27c31.png"><img width="1440" alt="af_issue_2" src="https://cloud.githubusercontent.com/assets/335167/14066262/c062e8b0-f3f9-11e5-80c9-ef1b09c27c31.png" style="max-width: 100%;"></a></li>
<li>after all retries exhausted non pooled subdag fails<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/335167/14066269/e1142484-f3f9-11e5-8355-454e0f892f7e.png"><img width="1440" alt="af_issue_6" src="https://cloud.githubusercontent.com/assets/335167/14066269/e1142484-f3f9-11e5-8355-454e0f892f7e.png" style="max-width: 100%;"></a></li>
<li>sub-task of pooled subdag fails<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/335167/14066263/d0251c8c-f3f9-11e5-9a32-7c92aeb50ced.png"><img width="1440" alt="af_issue_3" src="https://cloud.githubusercontent.com/assets/335167/14066263/d0251c8c-f3f9-11e5-9a32-7c92aeb50ced.png" style="max-width: 100%;"></a></li>
<li>pooled subdag remains in running state, sub-task of pooled subdag enters queued state<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/335167/14066264/d5d88218-f3f9-11e5-9772-e16a9f71bac4.png"><img width="1440" alt="af_issue_4" src="https://cloud.githubusercontent.com/assets/335167/14066264/d5d88218-f3f9-11e5-9772-e16a9f71bac4.png" style="max-width: 100%;"></a></li>
<li>sub-task of pooled subdag enters running state<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/335167/14066268/dbaa3b5a-f3f9-11e5-833a-df17e17235cd.png"><img width="1440" alt="af_issue_5" src="https://cloud.githubusercontent.com/assets/335167/14066268/dbaa3b5a-f3f9-11e5-833a-df17e17235cd.png" style="max-width: 100%;"></a></li>
<li>sub-task failures of pooled subdags generate email alerts for all failures<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/335167/14066271/e7c5a384-f3f9-11e5-9328-19bbffa7337f.png"><img width="1440" alt="af_issue_7" src="https://cloud.githubusercontent.com/assets/335167/14066271/e7c5a384-f3f9-11e5-9328-19bbffa7337f.png" style="max-width: 100%;"></a></li>
<li>logs (from UI) also show observed behavior</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Id  Dttm    Dag Id  Task Id Event   Execution Date  Owner   Extra
1910    03-27T15:37:36  test_18.test_subdag test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1911    03-27T15:37:46  test_18.test_subdag test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1916    03-27T15:39:18  test_18.test_subdag test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1918    03-27T15:39:28  test_18.test_subdag test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1924    03-27T15:41:18  test_18.test_subdag test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1930    03-27T15:41:28  test_18.test_subdag test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1912    03-27T15:37:54  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1914    03-27T15:38:05  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1917    03-27T15:39:23  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1920    03-27T15:39:33  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1921    03-27T15:40:54  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1922    03-27T15:41:04  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1925    03-27T15:41:26  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1926    03-27T15:41:36  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1927    03-27T15:41:52  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1928    03-27T15:42:02  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1929    03-27T15:42:18  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1931    03-27T15:42:28  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1933    03-27T15:43:52  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1934    03-27T15:44:02  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1935    03-27T15:44:30  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1936    03-27T15:44:40  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1937    03-27T15:44:56  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1938    03-27T15:45:06  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1939    03-27T15:46:35  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1940    03-27T15:46:45  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1941    03-27T15:47:42  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1942    03-27T15:47:52  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1943    03-27T15:49:20  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1944    03-27T15:49:30  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
..."><pre class="notranslate"><code class="notranslate">Id  Dttm    Dag Id  Task Id Event   Execution Date  Owner   Extra
1910    03-27T15:37:36  test_18.test_subdag test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1911    03-27T15:37:46  test_18.test_subdag test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1916    03-27T15:39:18  test_18.test_subdag test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1918    03-27T15:39:28  test_18.test_subdag test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1924    03-27T15:41:18  test_18.test_subdag test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1930    03-27T15:41:28  test_18.test_subdag test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1912    03-27T15:37:54  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1914    03-27T15:38:05  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1917    03-27T15:39:23  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1920    03-27T15:39:33  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1921    03-27T15:40:54  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1922    03-27T15:41:04  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1925    03-27T15:41:26  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1926    03-27T15:41:36  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1927    03-27T15:41:52  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1928    03-27T15:42:02  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1929    03-27T15:42:18  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1931    03-27T15:42:28  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1933    03-27T15:43:52  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1934    03-27T15:44:02  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1935    03-27T15:44:30  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1936    03-27T15:44:40  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1937    03-27T15:44:56  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1938    03-27T15:45:06  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1939    03-27T15:46:35  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1940    03-27T15:46:45  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1941    03-27T15:47:42  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1942    03-27T15:47:52  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
1943    03-27T15:49:20  test_18.test_subdag_with_pool   test_task_2_with_exit_1 running 03-26T12:00:00  Test    
1944    03-27T15:49:30  test_18.test_subdag_with_pool   test_task_2_with_exit_1 failed  03-26T12:00:00  Test    
...
</code></pre></div>
<ul dir="auto">
<li>Operating System: (Windows Version or <code class="notranslate">$ uname -a</code>)<br>
Linux **** 2.6.32-431.29.2.el6.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="69689814" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/1" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/1/hovercard" href="https://github.com/apache/airflow/pull/1">#1</a> SMP Tue Sep 9 21:36:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux</li>
<li>Python Version: <code class="notranslate">$ python --version</code><br>
Python 2.7.10</li>
</ul>
<p dir="auto">Now that you know a little about me, let me tell you about the issue I am having:</p>
<h1 dir="auto">Description of Issue</h1>
<p dir="auto">When subdag operators are used with pools, sub-task failures are not "bubbled" up to the subdag operator. The contained failed task will retry indefinitely even when retries is set to 0.</p>
<p dir="auto">When a subtask failure occurs the subdag operator remains in the running state, the sub-task quickly enters a queued state, and shortly there after re-enters a running state.</p>
<ul dir="auto">
<li>What did you expect to happen?</li>
<li>sub-task fails</li>
<li>subdag operator enters retry state</li>
<li>subdag operator enters running state when retry_delay has been met</li>
<li>sub-tasks get queued</li>
<li>sub-tasks enter running state</li>
<li>What happened instead?</li>
<li>sub-task fails</li>
<li>sub-task enters queued state</li>
<li>sub-task runs</li>
<li>rinse, repeat</li>
<li>Here is how you can reproduce this issue on your machine:
<h2 dir="auto">Reproduction Steps</h2>
</li>
<li>create a pool</li>
<li>set the email and pool in included DAG code</li>
<li>submit to your dags folder</li>
<li>observe non pooled subdag handle failure as specified</li>
<li>observe pooled subdag enter infinite loop</li>
</ul>
<p dir="auto">This behavior has been observed on multiple versions. I did the testing for submitting this issue on 1.7.0rc1.</p>
<p dir="auto">I also tested on HEAD 2016-03-26. The issue seems to get worst. The subdag which is not pooled (which works as expected on 1.7.0rc1) enters the retry state as it should ..  but then never re-enters a running state. Hanging indefinitely.</p>