<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.4.2</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">We upgraded airflow from 2.4.1 to 2.4.2 and immediately notice that every task log line is duplicated <em>into</em> CloudWatch. Comparing logs from tasks run before upgrade and after upgrade indicates that the issue is not in how the logs are displayed in Airflow, but rather that it now produces two log lines instead of one.</p>
<p dir="auto">When observing both the CloudWatch log streams and the Airflow UI, we can see duplicate log lines for <del><em>all</em></del> most log entries post upgrade, whilst seeing single log lines in tasks before upgrade.</p>
<p dir="auto">This happens <em>both</em> for tasks ran in a remote <code class="notranslate">EcsRunTaskOperator</code>'s as well as in regular <code class="notranslate">PythonOperator</code>'s.</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">A single non-duplicate log line should be produced into CloudWatch.</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">From my understanding now, any setup on 2.4.2 that uses CloudWatch remote logging will produce duplicate log lines. (But I have not been able to confirm other setups)</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Docker: <code class="notranslate">apache/airflow:2.4.2-python3.9</code> - Running on AWS ECS Fargate</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apache-airflow[celery,postgres,apache.hive,jdbc,mysql,ssh,amazon,google,google_auth]==2.4.2
apache-airflow-providers-amazon==6.0.0"><pre class="notranslate"><code class="notranslate">apache-airflow[celery,postgres,apache.hive,jdbc,mysql,ssh,amazon,google,google_auth]==2.4.2
apache-airflow-providers-amazon==6.0.0
</code></pre></div>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Other Docker-based deployment</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">We are running a docker inside Fargate ECS on AWS.</p>
<p dir="auto">The following environment variables + config in CloudFormation control remote logging:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="            - Name: AIRFLOW__LOGGING__REMOTE_LOGGING
              Value: True
            - Name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
              Value: !Sub &quot;cloudwatch://${TasksLogGroup.Arn}&quot;"><pre class="notranslate"><code class="notranslate">            - Name: AIRFLOW__LOGGING__REMOTE_LOGGING
              Value: True
            - Name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
              Value: !Sub "cloudwatch://${TasksLogGroup.Arn}"
</code></pre></div>
<h3 dir="auto">Anything else</h3>
<p dir="auto">We did not change any other configuration during the upgrade, simply bumped the requirements for provider list + docker image from 2.4.1 to 2.4.2.</p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<p dir="auto"><strong>Apache Airflow version</strong>:1.10.12</p>
<p dir="auto"><strong>Apache Airflow Provider versions</strong> (please include all providers that are relevant to your bug):<br>
No other providers</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):<br>
No using k8s</p>
<p dir="auto"><strong>Environment</strong>:<br>
AWS Linux</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):<br>
-NAME="Amazon Linux"<br>
VERSION="2"<br>
ID="amzn"<br>
ID_LIKE="centos rhel fedora"</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.14.186-146.268.amzn2.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="69689814" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/1" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/1/hovercard" href="https://github.com/apache/airflow/pull/1">#1</a> SMP Tue Jul 14 18:16:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: yum, pip3</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:<br>
I have been running airflow 1.10.12 for several months without any issues. Recently I updated airflow to use SSL cert. After that I reboot the airflow server and then restarted the airflow scheduler and Webserver as following:<br>
airflow Webserver &amp;<br>
airflow scheduler &amp;</p>
<p dir="auto">The jobs run normally at beginning. However, within one day, all the new jobs will immediately fail. From UI, I saw the task status as failed. However there is no log under the task when I checked the log from UI.</p>
<p dir="auto">I also checked the /logs/scheduler directory, except a warning of "psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "variable_key_key", which has existed for long time, I didn't see any other warning or error.</p>
<p dir="auto">Now this problem happened every day, i.e., if I restart airflow Webserver and scheduler, it works well. But within one day, all the new jobs will immediately fail with no log.</p>
<p dir="auto">Do you have any insight what can go wrong? Thanks.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong>:<br>
Not exactly sure how to reproduce. I just reboot airflow server.</p>
<p dir="auto"><strong>Anything else we need to know</strong>:</p>
<p dir="auto">How often does this problem occur? Once? Every time etc?<br>
I noticed the problem happens within 1 day.</p>
<p dir="auto">Any relevant logs to include? Put them here in side a detail tag:</p>