<p dir="auto"><strong>Apache Airflow version</strong>: 2.0.1</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):1.11</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: OpenShift 3.11</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): RHEL Server</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux</li>
<li><strong>Install tools</strong>: Airflow</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods "sales-trend-predict.181186911d0344eda3f3ff7d446f7d62" is forbidden: [maximum memory usage per Pod is 8Gi, but limit is 9663676416., minimum cpu usage per Container is 400m, but request is 1m., cpu max limit to request ratio per Container is 5, but provided ratio is 1000.000000.]","reason":"Forbidden","details":{"name":"sales-trend-predict.181186911d0344eda3f3ff7d446f7d62","kind":"pods"},"code":403}</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">For some enterprise kubernetes/openshift cluster, the admin will set a cpu/memory limit on container's resource, eg. minimum 400m cpu core, maximum 8Gi memory etc. However, for airflow's auto generated sidecar container for xcom 'airflow-xcom-sidecar' will fail this requirement as the error shown above, since the cpu core request is set to '1m' currently, (<a href="https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_generator.py#L60">https://github.com/apache/airflow/blob/master/airflow/kubernetes/pod_generator.py#L60</a>), as a result, the sidecar container won't be able to start up. Therefore, we need to have the capability to specify the xcom sidecar resource on kubernetesPodOperator as a parameter.</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>

<p dir="auto">This error only exist in certain kubernetes cluster that have a restriction on container's resource. Therefore, the error can be reproduced by manually setting a resource restriction on admin's level (eg. cpu core request 400m minimum, etc.). Then use KubernetesPodOperator with xcom enabled, you will be able to reproduce the error above.</p>
<p dir="auto"><strong>Anything else we need to know</strong>:<br>
N/A</p>
<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.1.4 (latest released)</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Debian GNU/Linux 10 (buster)</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto">apache-airflow-providers-amazon==2.2.0<br>
apache-airflow-providers-celery==2.0.0<br>
apache-airflow-providers-cncf-kubernetes==2.0.2<br>
apache-airflow-providers-docker==2.1.1<br>
apache-airflow-providers-elasticsearch==2.0.3<br>
apache-airflow-providers-ftp==2.0.1<br>
apache-airflow-providers-google==5.1.0<br>
apache-airflow-providers-grpc==2.0.1<br>
apache-airflow-providers-hashicorp==2.1.0<br>
apache-airflow-providers-http==2.0.1<br>
apache-airflow-providers-imap==2.0.1<br>
apache-airflow-providers-microsoft-azure==3.1.1<br>
apache-airflow-providers-mysql==2.1.1<br>
apache-airflow-providers-postgres==2.2.0<br>
apache-airflow-providers-redis==2.0.1<br>
apache-airflow-providers-sendgrid==2.0.1<br>
apache-airflow-providers-sftp==2.1.1<br>
apache-airflow-providers-slack==4.0.1<br>
apache-airflow-providers-sqlite==2.0.1<br>
apache-airflow-providers-ssh==2.1.1</p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Docker-Compose</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">dockerfile from apache/airflow:2.1.4-python3.7<br>
<code class="notranslate">EXECUTOR=LocalExecutor</code></p>
<h3 dir="auto">What happened</h3>
<p dir="auto">After successfully running a SUBDAG, clearing it (including downstream+recursive) doesn't trigger the inner tasks. Instead, the subdag is marked successful and the inner tasks all stay cleared and are not re-ran.</p>
<h3 dir="auto">What you expected to happen</h3>
<p dir="auto">Expected Clear with DownStream + Recursive to re-run all subdag tasks.</p>
<h3 dir="auto">How to reproduce</h3>
<ol dir="auto">
<li>Using a slightly modified version of <a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#subdags" rel="nofollow">https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#subdags</a>:</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from airflow import DAG
from airflow.example_dags.subdags.subdag import subdag
from airflow.operators.dummy import DummyOperator
from airflow.operators.subdag import SubDagOperator
from airflow.utils.dates import days_ago

def subdag(parent_dag_name, child_dag_name, args):
    dag_subdag = DAG(
        dag_id=f'{parent_dag_name}.{child_dag_name}',
        default_args=args,
        start_date=days_ago(2),
        schedule_interval=None,
    )

    for i in range(5):
        DummyOperator(
            task_id='{}-task-{}'.format(child_dag_name, i + 1),
            default_args=args,
            dag=dag_subdag,
        )

    return dag_subdag
	
DAG_NAME = 'example_subdag_operator'

args = {
    'owner': 'airflow',
}

dag = DAG(
    dag_id=DAG_NAME, default_args=args, start_date=days_ago(2), schedule_interval=None, tags=['example']
)

start = DummyOperator(
    task_id='start',
    dag=dag,
)

section_1 = SubDagOperator(
    task_id='section-1',
    subdag=subdag(DAG_NAME, 'section-1', args),
    dag=dag,
)

some_other_task = DummyOperator(
    task_id='some-other-task',
    dag=dag,
)

section_2 = SubDagOperator(
    task_id='section-2',
    subdag=subdag(DAG_NAME, 'section-2', args),
    dag=dag,
)

end = DummyOperator(
    task_id='end',
    dag=dag,
)

start &gt;&gt; section_1 &gt;&gt; some_other_task &gt;&gt; section_2 &gt;&gt; end"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">airflow</span> <span class="pl-k">import</span> <span class="pl-v">DAG</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">example_dags</span>.<span class="pl-s1">subdags</span>.<span class="pl-s1">subdag</span> <span class="pl-k">import</span> <span class="pl-s1">subdag</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">dummy</span> <span class="pl-k">import</span> <span class="pl-v">DummyOperator</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">subdag</span> <span class="pl-k">import</span> <span class="pl-v">SubDagOperator</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">dates</span> <span class="pl-k">import</span> <span class="pl-s1">days_ago</span>

<span class="pl-k">def</span> <span class="pl-en">subdag</span>(<span class="pl-s1">parent_dag_name</span>, <span class="pl-s1">child_dag_name</span>, <span class="pl-s1">args</span>):
    <span class="pl-s1">dag_subdag</span> <span class="pl-c1">=</span> <span class="pl-v">DAG</span>(
        <span class="pl-s1">dag_id</span><span class="pl-c1">=</span><span class="pl-s">f'<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">parent_dag_name</span><span class="pl-kos">}</span></span>.<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">child_dag_name</span><span class="pl-kos">}</span></span>'</span>,
        <span class="pl-s1">default_args</span><span class="pl-c1">=</span><span class="pl-s1">args</span>,
        <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-en">days_ago</span>(<span class="pl-c1">2</span>),
        <span class="pl-s1">schedule_interval</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,
    )

    <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">5</span>):
        <span class="pl-v">DummyOperator</span>(
            <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'{}-task-{}'</span>.<span class="pl-en">format</span>(<span class="pl-s1">child_dag_name</span>, <span class="pl-s1">i</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>),
            <span class="pl-s1">default_args</span><span class="pl-c1">=</span><span class="pl-s1">args</span>,
            <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag_subdag</span>,
        )

    <span class="pl-k">return</span> <span class="pl-s1">dag_subdag</span>
	
<span class="pl-v">DAG_NAME</span> <span class="pl-c1">=</span> <span class="pl-s">'example_subdag_operator'</span>

<span class="pl-s1">args</span> <span class="pl-c1">=</span> {
    <span class="pl-s">'owner'</span>: <span class="pl-s">'airflow'</span>,
}

<span class="pl-s1">dag</span> <span class="pl-c1">=</span> <span class="pl-v">DAG</span>(
    <span class="pl-s1">dag_id</span><span class="pl-c1">=</span><span class="pl-v">DAG_NAME</span>, <span class="pl-s1">default_args</span><span class="pl-c1">=</span><span class="pl-s1">args</span>, <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-en">days_ago</span>(<span class="pl-c1">2</span>), <span class="pl-s1">schedule_interval</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">tags</span><span class="pl-c1">=</span>[<span class="pl-s">'example'</span>]
)

<span class="pl-s1">start</span> <span class="pl-c1">=</span> <span class="pl-v">DummyOperator</span>(
    <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'start'</span>,
    <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
)

<span class="pl-s1">section_1</span> <span class="pl-c1">=</span> <span class="pl-v">SubDagOperator</span>(
    <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'section-1'</span>,
    <span class="pl-s1">subdag</span><span class="pl-c1">=</span><span class="pl-en">subdag</span>(<span class="pl-v">DAG_NAME</span>, <span class="pl-s">'section-1'</span>, <span class="pl-s1">args</span>),
    <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
)

<span class="pl-s1">some_other_task</span> <span class="pl-c1">=</span> <span class="pl-v">DummyOperator</span>(
    <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'some-other-task'</span>,
    <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
)

<span class="pl-s1">section_2</span> <span class="pl-c1">=</span> <span class="pl-v">SubDagOperator</span>(
    <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'section-2'</span>,
    <span class="pl-s1">subdag</span><span class="pl-c1">=</span><span class="pl-en">subdag</span>(<span class="pl-v">DAG_NAME</span>, <span class="pl-s">'section-2'</span>, <span class="pl-s1">args</span>),
    <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
)

<span class="pl-s1">end</span> <span class="pl-c1">=</span> <span class="pl-v">DummyOperator</span>(
    <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">'end'</span>,
    <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
)

<span class="pl-s1">start</span> <span class="pl-c1">&gt;&gt;</span> <span class="pl-s1">section_1</span> <span class="pl-c1">&gt;&gt;</span> <span class="pl-s1">some_other_task</span> <span class="pl-c1">&gt;&gt;</span> <span class="pl-s1">section_2</span> <span class="pl-c1">&gt;&gt;</span> <span class="pl-s1">end</span></pre></div>
<ol start="2" dir="auto">
<li>Run the subdag fully.</li>
<li>Clear (with recursive/downstream) any of the SubDags.</li>
<li>The Subdag will be marked successful, but if you zoom into the subdag, you'll see all the child tasks were not run.</li>
</ol>
<h3 dir="auto">Anything else</h3>
<p dir="auto">This appears to be the same issue as the "resolved" bug <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="774128101" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/13295" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/13295/hovercard" href="https://github.com/apache/airflow/issues/13295">#13295</a> . Airflow version 2.0.2 does not appear to fix the issue and neither does version 2.1.0 or 2.1.4. I am uncertain of how the fix (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="831154750" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/14776" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/14776/hovercard" href="https://github.com/apache/airflow/pull/14776">#14776</a>) was shown as the key to fixing this.</p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>