<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">main (development)</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">Variables with newlines in them are not always reproduced exactly as they are pasted into the web UI. Depending on how they are rendered, Unix newlines (<code class="notranslate">\n</code>) are silently converted to Windows newlines (<code class="notranslate">\r\n</code>) when rendered.</p>
<p dir="auto">Additionally, when users try to specify <code class="notranslate">\r</code> CR/"carriage return" in Python code, that gets rendered as a <code class="notranslate">\n</code> LF/"line feed" character.</p>
<p dir="auto">Tested on both astro and Airflow with Breeze.</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto"><del>Unix newlines (<code class="notranslate">\n</code>) should always be rendered as Unix newlines (<code class="notranslate">\n</code>); Windows newlines (<code class="notranslate">\r\n</code>) should always be rendered as Windows newlines.</del> Not 100% sure anymore.</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">Steps to reproduce:</p>
<ol dir="auto">
<li>Spin up Airflow with Breeze (<code class="notranslate">breeze start-airflow</code>) - I tested with the PostgreSQL database backend</li>
<li>Install <code class="notranslate">dos2unix</code> within the Airflow container: <code class="notranslate">apt update; apt install --yes dos2unix</code></li>
<li>Create a Variable in the Airflow UI named <code class="notranslate">newlines</code>, and ensure that its content contains newlines. Note that the non-newline content doesn't really matter. I used:
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Line 1
Line 2
Line 3"><pre class="notranslate"><code class="notranslate">Line 1
Line 2
Line 3
</code></pre></div>
</li>
<li>Run this dag:
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from datetime import datetime

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.smooth import SmoothOperator


with DAG(&quot;test_newlines&quot;, schedule=None) as dag:
    test_bash_operator_multiline_env_var = BashOperator(
        dag=dag,
        task_id=&quot;test_bash_operator_multiline_env_var&quot;,
        start_date=datetime.now(),
        env={
            &quot;MULTILINE_ENV_VAR&quot;: &quot;&quot;&quot;Line 1
Line 2
Line 3&quot;&quot;&quot;
        },
        append_env=True,
        bash_command='''
if [[ &quot;$(echo $MULTILINE_ENV_VAR)&quot; != &quot;$(echo $MULTILINE_ENV_VAR | dos2unix)&quot; ]]; then
    echo &gt;&amp;2 &quot;Multiline environment variable contains newlines incorrectly converted to Windows CRLF&quot;
    exit 1
fi''',
    )

    test_bash_operator_heredoc_contains_newlines = BashOperator(
        dag=dag,
        task_id=&quot;test_bash_operator_heredoc_contains_newlines&quot;,
        start_date=datetime.now(),
        bash_command=&quot;&quot;&quot;
diff &lt;(
cat &lt;&lt;EOF
Line 1
Line 2
Line 3
EOF
) &lt;(
cat &lt;&lt;EOF | dos2unix  # Hat-tip: @potiuk
Line 1
Line 2
Line 3
EOF
) || {
    echo &gt;&amp;2 &quot;Bash heredoc contains newlines incorrectly converted to Windows CRLF&quot;
    exit 1
}
&quot;&quot;&quot;,
    )

    test_bash_operator_env_var_from_variable_jinja_interpolation = BashOperator(
        dag=dag,
        task_id=&quot;test_bash_operator_env_var_from_variable_jinja_interpolation&quot;,
        start_date=datetime.now(),
        env={
            &quot;ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES&quot;: &quot;{{ var.value['newlines'] }}&quot;,
        },
        append_env=True,
        bash_command='''
diff &lt;(echo &quot;$ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES&quot;) &lt;(echo &quot;$ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES&quot; | dos2unix) || {
    echo &gt;&amp;2 &quot;Environment variable contains newlines incorrectly converted to Windows CRLF&quot;
    exit 1
}
''',
    )

    test_bash_operator_from_variable_jinja_interpolation = BashOperator(
        dag=dag,
        task_id=&quot;test_bash_operator_from_variable_jinja_interpolation&quot;,
        start_date=datetime.now(),
        bash_command='''
diff &lt;(echo &quot;{{ var.value['newlines'] }}&quot;) &lt;(echo &quot;{{ var.value['newlines'] }}&quot; | dos2unix) || {
    echo &gt;&amp;2 &quot;Jinja interpolated string contains newlines incorrectly converted to Windows CRLF&quot;
    exit 1
}
''',
    )

    test_bash_operator_backslash_n_not_equals_backslash_r = BashOperator(
        dag=dag,
        task_id=&quot;test_bash_operator_backslash_n_not_equals_backslash_r&quot;,
        start_date=datetime.now(),
        bash_command='''
if [[ &quot;\r&quot; == &quot;\n&quot; ]]; then
    echo &gt;&amp;2 &quot;Backslash-r has been incorrectly converted into backslash-n&quot;
    exit 1
fi''',
    )

    passing = SmoothOperator(dag=dag, task_id=&quot;passing&quot;, start_date=datetime.now())
    failing = SmoothOperator(dag=dag, task_id=&quot;failing&quot;, start_date=datetime.now())

    [
        test_bash_operator_multiline_env_var,
        test_bash_operator_heredoc_contains_newlines,
    ] &gt;&gt; passing
    [
        test_bash_operator_env_var_from_variable_jinja_interpolation,
        test_bash_operator_from_variable_jinja_interpolation,
        test_bash_operator_backslash_n_not_equals_backslash_r,
    ] &gt;&gt; failing"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">datetime</span>

<span class="pl-k">from</span> <span class="pl-s1">airflow</span> <span class="pl-k">import</span> <span class="pl-v">DAG</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">bash</span> <span class="pl-k">import</span> <span class="pl-v">BashOperator</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">operators</span>.<span class="pl-s1">smooth</span> <span class="pl-k">import</span> <span class="pl-v">SmoothOperator</span>


<span class="pl-k">with</span> <span class="pl-v">DAG</span>(<span class="pl-s">"test_newlines"</span>, <span class="pl-s1">schedule</span><span class="pl-c1">=</span><span class="pl-c1">None</span>) <span class="pl-k">as</span> <span class="pl-s1">dag</span>:
    <span class="pl-s1">test_bash_operator_multiline_env_var</span> <span class="pl-c1">=</span> <span class="pl-v">BashOperator</span>(
        <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"test_bash_operator_multiline_env_var"</span>,
        <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>(),
        <span class="pl-s1">env</span><span class="pl-c1">=</span>{
            <span class="pl-s">"MULTILINE_ENV_VAR"</span>: <span class="pl-s">"""Line 1</span>
<span class="pl-s">Line 2</span>
<span class="pl-s">Line 3"""</span>
        },
        <span class="pl-s1">append_env</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        <span class="pl-s1">bash_command</span><span class="pl-c1">=</span><span class="pl-s">'''</span>
<span class="pl-s">if [[ "$(echo $MULTILINE_ENV_VAR)" != "$(echo $MULTILINE_ENV_VAR | dos2unix)" ]]; then</span>
<span class="pl-s">    echo &gt;&amp;2 "Multiline environment variable contains newlines incorrectly converted to Windows CRLF"</span>
<span class="pl-s">    exit 1</span>
<span class="pl-s">fi'''</span>,
    )

    <span class="pl-s1">test_bash_operator_heredoc_contains_newlines</span> <span class="pl-c1">=</span> <span class="pl-v">BashOperator</span>(
        <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"test_bash_operator_heredoc_contains_newlines"</span>,
        <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>(),
        <span class="pl-s1">bash_command</span><span class="pl-c1">=</span><span class="pl-s">"""</span>
<span class="pl-s">diff &lt;(</span>
<span class="pl-s">cat &lt;&lt;EOF</span>
<span class="pl-s">Line 1</span>
<span class="pl-s">Line 2</span>
<span class="pl-s">Line 3</span>
<span class="pl-s">EOF</span>
<span class="pl-s">) &lt;(</span>
<span class="pl-s">cat &lt;&lt;EOF | dos2unix  # Hat-tip: @potiuk</span>
<span class="pl-s">Line 1</span>
<span class="pl-s">Line 2</span>
<span class="pl-s">Line 3</span>
<span class="pl-s">EOF</span>
<span class="pl-s">) || {</span>
<span class="pl-s">    echo &gt;&amp;2 "Bash heredoc contains newlines incorrectly converted to Windows CRLF"</span>
<span class="pl-s">    exit 1</span>
<span class="pl-s">}</span>
<span class="pl-s">"""</span>,
    )

    <span class="pl-s1">test_bash_operator_env_var_from_variable_jinja_interpolation</span> <span class="pl-c1">=</span> <span class="pl-v">BashOperator</span>(
        <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"test_bash_operator_env_var_from_variable_jinja_interpolation"</span>,
        <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>(),
        <span class="pl-s1">env</span><span class="pl-c1">=</span>{
            <span class="pl-s">"ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES"</span>: <span class="pl-s">"{{ var.value['newlines'] }}"</span>,
        },
        <span class="pl-s1">append_env</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        <span class="pl-s1">bash_command</span><span class="pl-c1">=</span><span class="pl-s">'''</span>
<span class="pl-s">diff &lt;(echo "$ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES") &lt;(echo "$ENV_VAR_AIRFLOW_VARIABLE_WITH_NEWLINES" | dos2unix) || {</span>
<span class="pl-s">    echo &gt;&amp;2 "Environment variable contains newlines incorrectly converted to Windows CRLF"</span>
<span class="pl-s">    exit 1</span>
<span class="pl-s">}</span>
<span class="pl-s">'''</span>,
    )

    <span class="pl-s1">test_bash_operator_from_variable_jinja_interpolation</span> <span class="pl-c1">=</span> <span class="pl-v">BashOperator</span>(
        <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"test_bash_operator_from_variable_jinja_interpolation"</span>,
        <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>(),
        <span class="pl-s1">bash_command</span><span class="pl-c1">=</span><span class="pl-s">'''</span>
<span class="pl-s">diff &lt;(echo "{{ var.value['newlines'] }}") &lt;(echo "{{ var.value['newlines'] }}" | dos2unix) || {</span>
<span class="pl-s">    echo &gt;&amp;2 "Jinja interpolated string contains newlines incorrectly converted to Windows CRLF"</span>
<span class="pl-s">    exit 1</span>
<span class="pl-s">}</span>
<span class="pl-s">'''</span>,
    )

    <span class="pl-s1">test_bash_operator_backslash_n_not_equals_backslash_r</span> <span class="pl-c1">=</span> <span class="pl-v">BashOperator</span>(
        <span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>,
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"test_bash_operator_backslash_n_not_equals_backslash_r"</span>,
        <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>(),
        <span class="pl-s1">bash_command</span><span class="pl-c1">=</span><span class="pl-s">'''</span>
<span class="pl-s">if [[ "<span class="pl-cce">\r</span>" == "<span class="pl-cce">\n</span>" ]]; then</span>
<span class="pl-s">    echo &gt;&amp;2 "Backslash-r has been incorrectly converted into backslash-n"</span>
<span class="pl-s">    exit 1</span>
<span class="pl-s">fi'''</span>,
    )

    <span class="pl-s1">passing</span> <span class="pl-c1">=</span> <span class="pl-v">SmoothOperator</span>(<span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>, <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"passing"</span>, <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>())
    <span class="pl-s1">failing</span> <span class="pl-c1">=</span> <span class="pl-v">SmoothOperator</span>(<span class="pl-s1">dag</span><span class="pl-c1">=</span><span class="pl-s1">dag</span>, <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"failing"</span>, <span class="pl-s1">start_date</span><span class="pl-c1">=</span><span class="pl-s1">datetime</span>.<span class="pl-en">now</span>())

    [
        <span class="pl-s1">test_bash_operator_multiline_env_var</span>,
        <span class="pl-s1">test_bash_operator_heredoc_contains_newlines</span>,
    ] <span class="pl-c1">&gt;&gt;</span> <span class="pl-s1">passing</span>
    [
        <span class="pl-s1">test_bash_operator_env_var_from_variable_jinja_interpolation</span>,
        <span class="pl-s1">test_bash_operator_from_variable_jinja_interpolation</span>,
        <span class="pl-s1">test_bash_operator_backslash_n_not_equals_backslash_r</span>,
    ] <span class="pl-c1">&gt;&gt;</span> <span class="pl-s1">failing</span></pre></div>
</li>
</ol>
<p dir="auto"><del>Warning: The <code class="notranslate">test_bash_operator_heredoc_contains_newlines</code> may not actually ever complete in Breeze. It does complete successfully when using <code class="notranslate">astro</code>. That might be another bug.</del> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/potiuk/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/potiuk">@potiuk</a>'s fix may have fixed this.</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">macOS 11.6.8 "Big Sur"</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Astronomer</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">Local breeze environment<br>
Local <code class="notranslate">astro</code> environment</p>
<h3 dir="auto">Anything else</h3>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<p dir="auto"><strong>Description</strong></p>
<p dir="auto">The old documentation for <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62697734" rel="nofollow">Contributor's guide</a> should be reviewed and incorporated in <a href="https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst">CONTRIBUTING.rst</a></p>
<p dir="auto"><strong>Use case / motivation</strong></p>
<p dir="auto">To remove documentation duplication and clean it up.</p>
<p dir="auto"><strong>Related Issues</strong></p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="673709470" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/10178" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/10178/hovercard" href="https://github.com/apache/airflow/issues/10178">#10178</a></p>