<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.3.0</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">Here i provide steps to reproduce this.</p>
<p dir="auto">Goal of this: to describe how to reproduce the "Failed to Adopt pod" error condition.</p>
<p dir="auto">The DAG-&gt;step Described Below should be of type KubernetesPodOperator</p>
<p dir="auto">NOTE: under normal operation,<br>
(where the MAIN_AIRFLOW_POD is never recycled by k8s, we will never see this edge-case)<br>
(it is only when the workerPod is still running, but the MAIN_AIRFLOW_POD is suddenly restarted/stopped)<br>
(that we would see orphan-&gt;workerPods)</p>
<p dir="auto">1] Implement a contrived-DAG, with a single step -&gt; which is long-running (e.g. 6 minutes)<br>
2] Deploy your <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-2">airflow-2</a>.1.4 / airfow-2.3.0 together with the contrived-DAG<br>
3] Run your contrived-DAG.<br>
4] in the middle of running the single-step, check via "kubectl" that your Kubernetes-&gt;workerPod has been created / running<br>
5] while workerPod still running, do "kubectl delete pod &lt;OF_MAIN_AIRFLOW_POD&gt;". This will mean that the workerPod becomes an orphan.<br>
6] the workerPod still continues to run through to completion. after which the K8S-&gt;status of the pod will be Completed, however the pod doesn't shut down itself.<br>
7] "kubectl" start up a new &lt;MAIN_AIRFLOW_POD&gt; so the web-ui is running again.<br>
8] MAIN_AIRFLOW_POD-&gt;webUi - Run your contrived-DAG again<br>
9] while the contrived-DAG is starting/tryingToStart etc, you will see in the logs printed out "Failed to adopt pod" -&gt; with 422 error code.</p>
<p dir="auto">The step-9 with the error message, you will find two appearances of this error msg in the <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-2">airflow-2</a>.1.4, <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-2">airflow-2</a>.3.0 source-code.<br>
The step-7 may also - general logging from the MAIN_APP - may also output the "Failed to adopt pod" error message also.</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">On previous versions of airflow e.g. 1.10.x, the orphan-workerPods would be adopted by the 2nd run-time of the airflowMainApp and either used to continue the same DAG and/or cleared away when complete.</p>
<p dir="auto">This is not happening with the newer airflow 2.1.4 / 2.3.0 (presumably because the code changed), and upon the 2nd run-time of the airflowMainApp - it would seem to try to adopt-workerPod but fails at that point ("Failed to adopt pod" in the logs and hence it cannot clear away orphan pods).</p>
<p dir="auto">Given this is an edge-case only, (i.e. we would not expect k8s to be recycling the main airflowApp/pod anyway), it doesn't seem totally urgent bug. However, the only reason for me raising this issue with yourselves is that given any k8s-&gt;namespace, in particular in PROD,   over time (e.g. 1 month?) the namespace will slowly be being filled up with orphanPods and somebody would need to manually log-in to delete old pods.</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">Here i provide steps to reproduce this.</p>
<p dir="auto">Goal of this: to describe how to reproduce the "Failed to Adopt pod" error condition.</p>
<p dir="auto">The DAG-&gt;step Described Below should be of type KubernetesPodOperator</p>
<p dir="auto">NOTE: under normal operation,<br>
(where the MAIN_AIRFLOW_POD is never recycled by k8s, we will never see this edge-case)<br>
(it is only when the workerPod is still running, but the MAIN_AIRFLOW_POD is suddenly restarted/stopped)<br>
(that we would see orphan-&gt;workerPods)</p>
<p dir="auto">1] Implement a contrived-DAG, with a single step -&gt; which is long-running (e.g. 6 minutes)<br>
2] Deploy your <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-2">airflow-2</a>.1.4 / airfow-2.3.0 together with the contrived-DAG<br>
3] Run your contrived-DAG.<br>
4] in the middle of running the single-step, check via "kubectl" that your Kubernetes-&gt;workerPod has been created / running<br>
5] while workerPod still running, do "kubectl delete pod &lt;OF_MAIN_AIRFLOW_POD&gt;". This will mean that the workerPod becomes an orphan.<br>
6] the workerPod still continues to run through to completion. after which the K8S-&gt;status of the pod will be Completed, however the pod doesn't shut down itself.<br>
7] "kubectl" start up a new &lt;MAIN_AIRFLOW_POD&gt; so the web-ui is running again.<br>
8] MAIN_AIRFLOW_POD-&gt;webUi - Run your contrived-DAG again<br>
9] while the contrived-DAG is starting/tryingToStart etc, you will see in the logs printed out "Failed to adopt pod" -&gt; with 422 error code.</p>
<p dir="auto">The step-9 with the error message, you will find two appearances of this error msg in the <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-2">airflow-2</a>.1.4, <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-2">airflow-2</a>.3.0 source-code.<br>
The step-7 may also - general logging from the MAIN_APP - may also output the "Failed to adopt pod" error message also.</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">kubernetes</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Other 3rd-party Helm chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">nothing special.</p>
<p dir="auto">it (CI/CD pipeline) builds the app. using requirements.txt to pull-in all the required python dependencies (including there is a dependency for the <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-2">airflow-2</a>.1.4 / 2.3.0)</p>
<p dir="auto">it (CI/CD pipeline) packages the app as an ECR image &amp; then deploy directly to k8s namespace.</p>
<h3 dir="auto">Anything else</h3>
<p dir="auto">this is 100% reproducible each &amp; every time.<br>
i have tested this multiple times.</p>
<p dir="auto">also - i tested this on the old <a class="issue-link js-issue-link notranslate" rel="noopener noreferrer nofollow" href="https://issues.apache.org/jira/browse/AIRFLOW-1">airflow-1</a>.10.x a couple of times to verify that the bug did not exist previously</p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<p dir="auto"><strong>Description</strong></p>
<p dir="auto">Add dependencies with "on-build" feature of Docker build.</p>
<p dir="auto"><strong>Use case / motivation</strong></p>
<p dir="auto">Some dependencies can be added easily using on-build feature of Docker build.</p>