<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.3.0 (latest released)</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">The unique ID of the task is seemingly not calculated correctly in base.py and it can cause in a duplicate key error, my guess is that this bit from /decorators/base.py:115:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if tg_task_id not in dag.task_ids:
    return task_id"><pre class="notranslate"><code class="notranslate">if tg_task_id not in dag.task_ids:
    return task_id
</code></pre></div>
<p dir="auto">doesn't make any sense and the actual return should be:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if tg_task_id not in dag.task_ids:
    return tg_task_id"><pre class="notranslate"><code class="notranslate">if tg_task_id not in dag.task_ids:
    return tg_task_id
</code></pre></div>
<p dir="auto">But I didn't write it so there is maybe a reason down the line for <code class="notranslate">return task_id</code>. Someone that does know his way around TaskFlow's codebase should pitch in.</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">task.expand should not lead to a airflow.exceptions.DuplicateTaskIdFound inside task groups.</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">Pick example_task_group_decorator.py, change it so that task_1 returns a list, and call task_2 with <code class="notranslate">.expand(value=task_1)</code>.</p>
<p dir="auto">A modification as simple as this will already kill the DAG:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# &quot;License&quot;); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# &quot;AS IS&quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

&quot;&quot;&quot;Example DAG demonstrating the usage of the @taskgroup decorator.&quot;&quot;&quot;

import pendulum

from airflow.decorators import task, task_group
from airflow.models.dag import DAG


# [START howto_task_group_decorator]
# Creating Tasks
@task
def task_start():
    &quot;&quot;&quot;Empty Task which is First Task of Dag&quot;&quot;&quot;
    return '[Task_start]'


@task
def task_1(value: int) -&gt; list[str]:
    &quot;&quot;&quot;Empty Task1&quot;&quot;&quot;
    return [f'[ Task1 {value} ]', f'[ Task1 {value+1} ]']


@task
def task_2(value: list[str]) -&gt; str:
    &quot;&quot;&quot;Empty Task2&quot;&quot;&quot;
    return f'[ Task2 {value} ]'


@task
def task_3(value: str) -&gt; None:
    &quot;&quot;&quot;Empty Task3&quot;&quot;&quot;
    print(f'[ Task3 {value} ]')


@task
def task_end() -&gt; None:
    &quot;&quot;&quot;Empty Task which is Last Task of Dag&quot;&quot;&quot;
    print('[ Task_End  ]')


# Creating TaskGroups
@task_group
def task_group_function(value: int) -&gt; None:
    &quot;&quot;&quot;TaskGroup for grouping related Tasks&quot;&quot;&quot;
    task_3(task_2.expand(value=task_1(value)))


# Executing Tasks and TaskGroups
with DAG(
    dag_id=&quot;example_task_group_decorator&quot;,
    start_date=pendulum.datetime(2021, 1, 1, tz=&quot;UTC&quot;),
    catchup=False,
    tags=[&quot;example&quot;],
) as dag:
    start_task = task_start()
    end_task = task_end()
    for i in range(5):
        current_task_group = task_group_function(i)
        start_task &gt;&gt; current_task_group &gt;&gt; end_task

# [END howto_task_group_decorator]"><pre class="notranslate"><code class="notranslate">#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

"""Example DAG demonstrating the usage of the @taskgroup decorator."""

import pendulum

from airflow.decorators import task, task_group
from airflow.models.dag import DAG


# [START howto_task_group_decorator]
# Creating Tasks
@task
def task_start():
    """Empty Task which is First Task of Dag"""
    return '[Task_start]'


@task
def task_1(value: int) -&gt; list[str]:
    """Empty Task1"""
    return [f'[ Task1 {value} ]', f'[ Task1 {value+1} ]']


@task
def task_2(value: list[str]) -&gt; str:
    """Empty Task2"""
    return f'[ Task2 {value} ]'


@task
def task_3(value: str) -&gt; None:
    """Empty Task3"""
    print(f'[ Task3 {value} ]')


@task
def task_end() -&gt; None:
    """Empty Task which is Last Task of Dag"""
    print('[ Task_End  ]')


# Creating TaskGroups
@task_group
def task_group_function(value: int) -&gt; None:
    """TaskGroup for grouping related Tasks"""
    task_3(task_2.expand(value=task_1(value)))


# Executing Tasks and TaskGroups
with DAG(
    dag_id="example_task_group_decorator",
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example"],
) as dag:
    start_task = task_start()
    end_task = task_end()
    for i in range(5):
        current_task_group = task_group_function(i)
        start_task &gt;&gt; current_task_group &gt;&gt; end_task

# [END howto_task_group_decorator]
</code></pre></div>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Manjaro Linux</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Virtualenv installation</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<p dir="auto">I wanted to discuss an important change proposal for our Breeze2 approach.</p>
<p dir="auto">TL;DR; I thought a lot about how we shold run Breeze2. Instead of having a single <code class="notranslate">./breeze</code> command we could have multiple <code class="notranslate">airflow-*</code> commands that would be automatically installed by <code class="notranslate">pipx install -e ./dev/breeze</code> and that it should be the only way of running various commands of breeze (both in CI and during local development). An example of that is <code class="notranslate">airflow-freespace</code> command that is used in CI to free space needed to run the commands.</p>
<h1 dir="auto">Context</h1>
<p dir="auto">I looked at some of the changes made recently, particularly just merged: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1094932287" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/20701" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/20701/hovercard" href="https://github.com/apache/airflow/pull/20701">#20701</a> and I thought we might change the approach of the new Breeze following the learnings we had and mostly suggestions of <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/uranusjr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/uranusjr">@uranusjr</a>  when we dicussed the custom code in <a href="https://github.com/apache/airflow/blob/main/Breeze2">https://github.com/apache/airflow/blob/main/Breeze2</a> for bootstraping the environment.</p>
<p dir="auto">This is - in essence very similar aas the old <code class="notranslate">./breeze</code> was and I kept to be "anchored" with it however by introducing <code class="notranslate">pipx</code> and installing ./breeze locally in editable mode, we've almost achieved what I wanted from the beginning - i.e. very simple and os-independent way of bootstrapping the environment for Breeze.</p>
<p dir="auto">Current state, reasoning  and decision is captured here (still Draft though): <a href="https://github.com/apache/airflow/blob/main/dev/breeze/doc/adr/0003-bootstraping-virtual-environment.md">https://github.com/apache/airflow/blob/main/dev/breeze/doc/adr/0003-bootstraping-virtual-environment.md</a></p>
<h1 dir="auto">Proposal</h1>
<p dir="auto">The "free-space" script shows how we can utilize multiple entry-points,. Instead of "one-command to rule them all" - "Breeze"  - we could have multiple commands/entry-points installed with <code class="notranslate">pipx</code>:</p>
<ul dir="auto">
<li>airflow-breeze -&gt; to run Breeze shell</li>
<li>airflow-build-ci-image -&gt; to build ci image</li>
<li>airflow-build-prod-image-&gt; to build prod image</li>
<li>airflow-build-docs - &gt; to build docs,</li>
<li>airflow-run-static-checks - &gt; to run static checks</li>
</ul>
<p dir="auto">and so on, rather than:</p>
<ul dir="auto">
<li>./Breeze2 build-ci-image</li>
<li>./Breeze2 build-docs</li>
</ul>
<p dir="auto">and so on.</p>
<h1 dir="auto">Why it might be better ?</h1>
<p dir="auto">Mainly because of <code class="notranslate">zen of python</code>: <a href="https://www.python.org/dev/peps/pep-0020/" rel="nofollow">https://www.python.org/dev/peps/pep-0020/</a></p>
<blockquote>
<p dir="auto">There should be one-- and preferably only one --obvious way to do it.</p>
</blockquote>
<p dir="auto">Wile <code class="notranslate">pipx-only</code> approach is not perfect, I think we can make it close to it (with some effort and added complexity).</p>
<p dir="auto">With PIPX installation, all those commands are easily accessible in the PATH (and auto-complete'able).</p>
<p dir="auto">Also the very same commands could be used in CI (same as we did with "airflow-freespace" - this would limit some duplication and made it much more straightforward - how to run some commands locally. The parameters for those commands could still be passed by environment variables in CI (Click has support for that and we already use it).</p>
<p dir="auto">The only problem to solve is how to handle errors when somene has "old"  "breeze" virtualenv. Breeze will change dynamically - new dependencies will be added and we will move to newer versions regularly. This might lead to unnecessary issues and  questions on Slack if those errorrs are somehow mysterious and not obvious. Current ./Breeze2 bootstrap script does it by automated upgrading of the virtualenv. However I thought about it and maybe we can approach it differently.</p>
<p dir="auto">I think we have a way to handle it. We should write a little common code for those commands to capture and store in repo what dependencies are needed (and which versions - that could be even requirements.txt file) and update it automatically with pre-commit and warn the users in cases the dependencies are missing or outdated whenever any command is run. Those dependencies will not change often, but when they do, all users should know that they should upgrade and it should be possible.</p>
<p dir="auto">This would be a warning only but it could provide a helpful "Your breeze enviroinment is outdated, please run <code class="notranslate">pipx install -e ./dev/breeze --force</code>  to update the environment" warning.</p>
<p dir="auto">Then we could (as <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/uranusjr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/uranusjr">@uranusjr</a> proposed and stronly suggested) remove the custom <code class="notranslate">./Breeze2</code> bootstrapping.</p>
<h1 dir="auto">Why it might be worse ?</h1>
<p dir="auto">There are two serious dangers with pipx:</p>
<ul dir="auto">
<li>that the version you have in one branch will not work with another branch (very likely from the past experiences)</li>
<li>that even if you are in the same branch, you could have not installed the latest version of dependencies (and Breeze will fail) or that you have not installed it with --editable flag - which means that the code of Breeze in the branch is different that what you run. This might lead to serious, unforeseen and difficult to diagnose problems.</li>
</ul>
<p dir="auto">Both are real dangers, and they will significantly increase the likelihood that different users (especially the ones who are not experienced) will get Breeze not working and that they will raise support questions or (worse) that they will consider Breeze as buggy and will avoid using it or even will drop the idea of contributing to Airflow.</p>
<p dir="auto">The ./Breeze2 bootstrapping is free of those problems:</p>
<ul dir="auto">
<li>--editable mode is automatically used</li>
<li>latest dependencies are added when they were missing automatically</li>
<li>your Breeze version is very tightly connecteect to the branch you are in</li>
</ul>