<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.5.0</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">I have a dynamic mapping task that is supposed to launch over 100 KubernetesPodOperator tasks. I have assigned 2.0 CPUs per task. When running the DAG, 16 tasks are in 'running state', however only 3 truly run, the remainder 13 fail with <code class="notranslate">Pod took longer than 120 seconds to start</code>. The remainder of the tasks are either queued or scheduled, and when there are less than 16 active tasks, they run and more or less fail with the same error.</p>
<p dir="auto">Here is a snapshot of</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl -n airflow get all
NAME                                                            READY   STATUS              RESTARTS   AGE
pod/airflow-postgresql-0                                        1/1     Running             5          2d
pod/airflow-scheduler-6dd68b485c-w8bhp                          3/3     Running             19         2d
pod/airflow-statsd-586dbdcc6b-h4mnr                             1/1     Running             5          2d
pod/airflow-triggerer-95565b95d-phts7                           2/2     Running             14         2d
pod/airflow-webserver-599bb95bcd-7dtpk                          1/1     Running             5          2d
pod/my-task-17dd038ca4d04164ba90f9c7f9a7fbb6            0/2     Pending             0          49s
pod/my-task-20aba86c65544ea384343f8fb4415d3a            0/2     Pending             0          53s
pod/my-task-3c5b4444a7d242459907ff3be7b7d6f6            0/2     Pending             0          44s
pod/my-task-5c8af5edb0904711b6a76a2edf1d1067            0/2     Pending             0          60s
pod/my-task-6001d3567f96400bb0ae559f22d3d2db            0/2     Pending             0          43s
pod/my-task-6dfb1945f3ff4ac4a06c7e6c6a85099c            0/2     Pending             0          81s
pod/my-task-71ad2fb48fb64f449014bba45bee980f            0/2     ContainerCreating   0          52s
pod/my-task-774216cb5f9344ffb35deac826d71639            0/2     Pending             0          68s
pod/my-task-814266d425254130868c3a5ebc8dce49            0/2     Pending             0          67s
pod/my-task-a11588d878b54944b4c069f49231ac36            0/2     Pending             0          77s
pod/my-task-b16c843fa038441ea31b90363ed86aa0            0/2     Pending             0          49s
pod/my-task-b85e2ed3417a4a62940661f418c900e5            0/2     Pending             0          60s
pod/my-task-d1de2a771a104a2592956a713f785300            0/2     Pending             0          73s
pod/my-task-dbeba55a80074c08bbdf023b3f0b885c            0/2     Completed           0          10m
pod/my-task-f83ad2805d314be3a7307b7216a54e53            2/2     Running             0          10m
pod/pipeline-my-task-0bc9e094afee4527b5b764e32f590282   0/1     Init:0/1            0          1s
pod/pipeline-my-task-1d51c5d3776e4dd8a89461e8a76faba1   1/1     Running             0          62s
pod/pipeline-my-task-24b1326a71d149fb9f62c101647468ee   1/1     Running             0          62s
pod/pipeline-my-task-29b132b7b0ce4832a5e30a821c6405bf   1/1     Running             0          10m
pod/pipeline-my-task-29fb55604eec457fa21d13d85c7889b5   1/1     Running             0          10m
pod/pipeline-my-task-2a337f1cc28b4315945cec8a961b1111   1/1     Running             0          69s
pod/pipeline-my-task-35d5c97570474082bc9b04189c433be7   1/1     Running             0          57s
pod/pipeline-my-task-569de133975d4dbb96becb2a04c0dac3   1/1     Running             0          78s
pod/pipeline-my-task-96a9681ace4441deba4faeef602f6e5b   1/1     Running             0          78s
pod/pipeline-my-task-9dcb9578720643eca5fa918a0a295f87   1/1     Running             0          87s
pod/pipeline-my-task-a643741d29ea4f4baa06e0ea20bc1a57   1/1     Running             0          10m
pod/pipeline-my-task-b04532a9f35a48a09cb1d46c9d9470dd   1/1     Running             0          57s
pod/pipeline-my-task-c9b7bb4ee07749be98083a11a512e1f4   1/1     Running             0          90s
pod/pipeline-my-task-d9c5ce9bf5ce499583cdf0ea3f58b7f0   1/1     Running             0          82s
pod/pipeline-my-task-dd5a5a45374f487fbc34c904e71b93b5   1/1     Running             0          59s
pod/pipeline-my-task-ea8c39d657824a1db505b00e8673b06a   1/1     Running             0          69s
pod/pipeline-my-task-fb5c71d274034f5392aebe0f4b395d98   1/1     Running             0          65s"><pre class="notranslate"><code class="notranslate">kubectl -n airflow get all
NAME                                                            READY   STATUS              RESTARTS   AGE
pod/airflow-postgresql-0                                        1/1     Running             5          2d
pod/airflow-scheduler-6dd68b485c-w8bhp                          3/3     Running             19         2d
pod/airflow-statsd-586dbdcc6b-h4mnr                             1/1     Running             5          2d
pod/airflow-triggerer-95565b95d-phts7                           2/2     Running             14         2d
pod/airflow-webserver-599bb95bcd-7dtpk                          1/1     Running             5          2d
pod/my-task-17dd038ca4d04164ba90f9c7f9a7fbb6            0/2     Pending             0          49s
pod/my-task-20aba86c65544ea384343f8fb4415d3a            0/2     Pending             0          53s
pod/my-task-3c5b4444a7d242459907ff3be7b7d6f6            0/2     Pending             0          44s
pod/my-task-5c8af5edb0904711b6a76a2edf1d1067            0/2     Pending             0          60s
pod/my-task-6001d3567f96400bb0ae559f22d3d2db            0/2     Pending             0          43s
pod/my-task-6dfb1945f3ff4ac4a06c7e6c6a85099c            0/2     Pending             0          81s
pod/my-task-71ad2fb48fb64f449014bba45bee980f            0/2     ContainerCreating   0          52s
pod/my-task-774216cb5f9344ffb35deac826d71639            0/2     Pending             0          68s
pod/my-task-814266d425254130868c3a5ebc8dce49            0/2     Pending             0          67s
pod/my-task-a11588d878b54944b4c069f49231ac36            0/2     Pending             0          77s
pod/my-task-b16c843fa038441ea31b90363ed86aa0            0/2     Pending             0          49s
pod/my-task-b85e2ed3417a4a62940661f418c900e5            0/2     Pending             0          60s
pod/my-task-d1de2a771a104a2592956a713f785300            0/2     Pending             0          73s
pod/my-task-dbeba55a80074c08bbdf023b3f0b885c            0/2     Completed           0          10m
pod/my-task-f83ad2805d314be3a7307b7216a54e53            2/2     Running             0          10m
pod/pipeline-my-task-0bc9e094afee4527b5b764e32f590282   0/1     Init:0/1            0          1s
pod/pipeline-my-task-1d51c5d3776e4dd8a89461e8a76faba1   1/1     Running             0          62s
pod/pipeline-my-task-24b1326a71d149fb9f62c101647468ee   1/1     Running             0          62s
pod/pipeline-my-task-29b132b7b0ce4832a5e30a821c6405bf   1/1     Running             0          10m
pod/pipeline-my-task-29fb55604eec457fa21d13d85c7889b5   1/1     Running             0          10m
pod/pipeline-my-task-2a337f1cc28b4315945cec8a961b1111   1/1     Running             0          69s
pod/pipeline-my-task-35d5c97570474082bc9b04189c433be7   1/1     Running             0          57s
pod/pipeline-my-task-569de133975d4dbb96becb2a04c0dac3   1/1     Running             0          78s
pod/pipeline-my-task-96a9681ace4441deba4faeef602f6e5b   1/1     Running             0          78s
pod/pipeline-my-task-9dcb9578720643eca5fa918a0a295f87   1/1     Running             0          87s
pod/pipeline-my-task-a643741d29ea4f4baa06e0ea20bc1a57   1/1     Running             0          10m
pod/pipeline-my-task-b04532a9f35a48a09cb1d46c9d9470dd   1/1     Running             0          57s
pod/pipeline-my-task-c9b7bb4ee07749be98083a11a512e1f4   1/1     Running             0          90s
pod/pipeline-my-task-d9c5ce9bf5ce499583cdf0ea3f58b7f0   1/1     Running             0          82s
pod/pipeline-my-task-dd5a5a45374f487fbc34c904e71b93b5   1/1     Running             0          59s
pod/pipeline-my-task-ea8c39d657824a1db505b00e8673b06a   1/1     Running             0          69s
pod/pipeline-my-task-fb5c71d274034f5392aebe0f4b395d98   1/1     Running             0          65s
</code></pre></div>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">Only 3 tasks should be running.<br>
The remainder tasks should be scheduled or queued.</p>
<h3 dir="auto">How to reproduce</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="
import json
import textwrap

import pendulum

from airflow.decorators import dag, task
from airflow.models.param import Param
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
    Secret,
)
from kubernetes.client import models as k8s


@dag(
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz=&quot;UTC&quot;),
    catchup=False,
    tags=[&quot;example&quot;],
)
def pipeline():

    container_resources = k8s.V1ResourceRequirements(
        limits={
            &quot;memory&quot;: &quot;512Mi&quot;,
            &quot;cpu&quot;: 2.0,
        },
        requests={
            &quot;memory&quot;: &quot;512Mi&quot;,
            &quot;cpu&quot;: 2.0,
        },
    )


    volumes = [
        k8s.V1Volume(
            name=&quot;pvc-airflow&quot;,
            persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(
                claim_name=&quot;pvc-airflow&quot;
            ),
        )
    ]

    volume_mounts = [
        k8s.V1VolumeMount(mount_path=&quot;/airflow&quot;, name=&quot;pvc-airflow&quot;, sub_path=None)
    ]

    @task
    def make_list():
        return [{&quot;a&quot;: &quot;a&quot;}] * 100

    my_task = KubernetesPodOperator.partial(
        name=&quot;my_task&quot;,
        task_id=&quot;my_task&quot;,
        image=&quot;ubuntu:20.04&quot;,
        namespace=&quot;airflow&quot;,
        container_resources=container_resources,
        volumes=volumes,
        volume_mounts=volume_mounts,
        in_cluster=True,
        do_xcom_push=True,
        get_logs=True,
        cmds=[
            &quot;/bin/bash&quot;,
            &quot;-c&quot;,
            &quot;&quot;&quot;
                sleep 600
            &quot;&quot;&quot;
        ],
    ).expand(env_vars=make_list())"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">json</span>
<span class="pl-k">import</span> <span class="pl-s1">textwrap</span>

<span class="pl-k">import</span> <span class="pl-s1">pendulum</span>

<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">decorators</span> <span class="pl-k">import</span> <span class="pl-s1">dag</span>, <span class="pl-s1">task</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">models</span>.<span class="pl-s1">param</span> <span class="pl-k">import</span> <span class="pl-v">Param</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">providers</span>.<span class="pl-s1">cncf</span>.<span class="pl-s1">kubernetes</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">kubernetes_pod</span> <span class="pl-k">import</span> (
    <span class="pl-v">KubernetesPodOperator</span>,
    <span class="pl-v">Secret</span>,
)
<span class="pl-k">from</span> <span class="pl-s1">kubernetes</span>.<span class="pl-s1">client</span> <span class="pl-k">import</span> <span class="pl-s1">models</span> <span class="pl-k">as</span> <span class="pl-s1">k8s</span>


<span class="pl-en">@<span class="pl-en">dag</span>(</span>
<span class="pl-en">    <span class="pl-s1">schedule</span><span class="pl-c1">=</span><span class="pl-c1">None</span>,</span>
<span class="pl-en">    <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">pendulum</span>.<span class="pl-en">datetime</span>(<span class="pl-c1">2021</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-s1">tz</span><span class="pl-c1">=</span><span class="pl-s">"UTC"</span>),</span>
<span class="pl-en">    <span class="pl-s1">catchup</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,</span>
<span class="pl-en">    <span class="pl-s1">tags</span><span class="pl-c1">=</span>[<span class="pl-s">"example"</span>],</span>
<span class="pl-en">)</span>
<span class="pl-k">def</span> <span class="pl-en">pipeline</span>():

    <span class="pl-s1">container_resources</span> <span class="pl-c1">=</span> <span class="pl-s1">k8s</span>.<span class="pl-v">V1ResourceRequirements</span>(
        <span class="pl-s1">limits</span><span class="pl-c1">=</span>{
            <span class="pl-s">"memory"</span>: <span class="pl-s">"512Mi"</span>,
            <span class="pl-s">"cpu"</span>: <span class="pl-c1">2.0</span>,
        },
        <span class="pl-s1">requests</span><span class="pl-c1">=</span>{
            <span class="pl-s">"memory"</span>: <span class="pl-s">"512Mi"</span>,
            <span class="pl-s">"cpu"</span>: <span class="pl-c1">2.0</span>,
        },
    )


    <span class="pl-s1">volumes</span> <span class="pl-c1">=</span> [
        <span class="pl-s1">k8s</span>.<span class="pl-v">V1Volume</span>(
            <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"pvc-airflow"</span>,
            <span class="pl-s1">persistent_volume_claim</span><span class="pl-c1">=</span><span class="pl-s1">k8s</span>.<span class="pl-v">V1PersistentVolumeClaimVolumeSource</span>(
                <span class="pl-s1">claim_name</span><span class="pl-c1">=</span><span class="pl-s">"pvc-airflow"</span>
            ),
        )
    ]

    <span class="pl-s1">volume_mounts</span> <span class="pl-c1">=</span> [
        <span class="pl-s1">k8s</span>.<span class="pl-v">V1VolumeMount</span>(<span class="pl-s1">mount_path</span><span class="pl-c1">=</span><span class="pl-s">"/airflow"</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"pvc-airflow"</span>, <span class="pl-s1">sub_path</span><span class="pl-c1">=</span><span class="pl-c1">None</span>)
    ]

    <span class="pl-en">@<span class="pl-s1">task</span></span>
    <span class="pl-k">def</span> <span class="pl-en">make_list</span>():
        <span class="pl-k">return</span> [{<span class="pl-s">"a"</span>: <span class="pl-s">"a"</span>}] <span class="pl-c1">*</span> <span class="pl-c1">100</span>

    <span class="pl-s1">my_task</span> <span class="pl-c1">=</span> <span class="pl-v">KubernetesPodOperator</span>.<span class="pl-en">partial</span>(
        <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"my_task"</span>,
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"my_task"</span>,
        <span class="pl-s1">image</span><span class="pl-c1">=</span><span class="pl-s">"ubuntu:20.04"</span>,
        <span class="pl-s1">namespace</span><span class="pl-c1">=</span><span class="pl-s">"airflow"</span>,
        <span class="pl-s1">container_resources</span><span class="pl-c1">=</span><span class="pl-s1">container_resources</span>,
        <span class="pl-s1">volumes</span><span class="pl-c1">=</span><span class="pl-s1">volumes</span>,
        <span class="pl-s1">volume_mounts</span><span class="pl-c1">=</span><span class="pl-s1">volume_mounts</span>,
        <span class="pl-s1">in_cluster</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        <span class="pl-s1">do_xcom_push</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        <span class="pl-s1">get_logs</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        <span class="pl-s1">cmds</span><span class="pl-c1">=</span>[
            <span class="pl-s">"/bin/bash"</span>,
            <span class="pl-s">"-c"</span>,
            <span class="pl-s">"""</span>
<span class="pl-s">                sleep 600</span>
<span class="pl-s">            """</span>
        ],
    ).<span class="pl-en">expand</span>(<span class="pl-s1">env_vars</span><span class="pl-c1">=</span><span class="pl-en">make_list</span>())</pre></div>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Ubuntu 20.04.5 LTS</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Official Apache Airflow Helm Chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">I am running this locally using the helm chart on Kind.</p>
<p dir="auto">My machine is 4 CPU (x2), with 16 GB RAM.</p>
<h3 dir="auto">Anything else</h3>
<p dir="auto">I have confirmed that the failing tasks are not starting due to timeouts from waiting for resources too long.</p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<p dir="auto">The tests that are run in CI should use <code class="notranslate">breeze</code> and parallellism implemented in Python.</p>
<p dir="auto">NOTE: We should pay an attention to provide better error feedback when docker containers could not be started - see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1227699054" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/23523" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/23523/hovercard" href="https://github.com/apache/airflow/issues/23523">#23523</a></p>