<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">Other Airflow 2 version</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">We noticed that tasks using the dynamic task mapping api crashes the scheduler with the error</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint &quot;task_instance_pkey&quot;
80
DETAIL:  Key (dag_id, task_id, run_id, map_index)=(ingestion-scheduler, check_for_success_flag, manual__2022-10-03T15:56:03.241451+00:00, 0) already exists."><pre class="notranslate"><code class="notranslate">psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "task_instance_pkey"
80
DETAIL:  Key (dag_id, task_id, run_id, map_index)=(ingestion-scheduler, check_for_success_flag, manual__2022-10-03T15:56:03.241451+00:00, 0) already exists.
</code></pre></div>
<p dir="auto">We have 2 schedulers so my guess is that both try to schedule the dynamically mapped task</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">it should be not crash the scheduler</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">run a large dynamically mapped task in multi-scheduler environment</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">linux</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Other Docker-based deployment</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.2.4 (latest released)</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">Documentation for <a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/kubernetes_pod/index.html#airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator" rel="nofollow"><code class="notranslate">airflow.providers.cncf.kubernetes.operators.kubernetes_pod</code></a> suggests that <a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/kubernetes_pod/index.html#airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator.template_fields" rel="nofollow"><code class="notranslate">template_fields</code></a> are a list of string values. This is accordance with Airflow's custom operator guides, to create a class attribute with a list of string values. However, when consulting the <a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_modules/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.html#KubernetesPodOperator.template_fields" rel="nofollow">source code</a>, the values for <code class="notranslate">template_fields</code> are stored in a tuple:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="template_fields: Sequence[str] = (
        'image',
        'cmds',
        'arguments',
        'env_vars',
        'labels',
        'config_file',
        'pod_template_file',
        'namespace',

    )"><pre class="notranslate"><code class="notranslate">template_fields: Sequence[str] = (
        'image',
        'cmds',
        'arguments',
        'env_vars',
        'labels',
        'config_file',
        'pod_template_file',
        'namespace',

    )
</code></pre></div>
<p dir="auto">This is problematic from a documentation standpoint, since it is unclear how to extend behaviors with ambiguous typing. For example, I will encounter a runtime error when extending template fields as such:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

class CustomKubernetesPodOperator(KubernetesPodOperator):
    # The following will fail
    template_fields = KubernetesPodOperator.template_fields + [&quot;name&quot;]

    def __init__(self):
        super.__init__()"><pre class="notranslate"><code class="notranslate">from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

class CustomKubernetesPodOperator(KubernetesPodOperator):
    # The following will fail
    template_fields = KubernetesPodOperator.template_fields + ["name"]

    def __init__(self):
        super.__init__()
</code></pre></div>
<p dir="auto">My solution is to wrap the template_fields in a <code class="notranslate">list</code> constructor as such:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="template_fields = list(KubernetesPodOperator.template_fields) + [&quot;name&quot;]"><pre class="notranslate"><code class="notranslate">template_fields = list(KubernetesPodOperator.template_fields) + ["name"]
</code></pre></div>
<p dir="auto">But this is not an elegant solution. Plus, documentation was not helpful. A solution would be to either clarify the documentation (that this is stored as a tuple default), or convert to storing as a list. (As typical with other Airflow operators.)</p>
<h3 dir="auto">What you expected to happen</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">NA</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Other</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">NA</p>
<h3 dir="auto">Anything else</h3>
<p dir="auto">NA</p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>