<h1 dir="auto">Why Mypy re-enable</h1>
<p dir="auto">For a few weeks MyPy checks have been disabled after the switch to Python 3.7 (per <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1039799030" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/19317" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/19317/hovercard" href="https://github.com/apache/airflow/pull/19317">#19317</a>).</p>
<p dir="auto">We should, however, re-enable it back as it is very useful in catching a number of mistakes.</p>
<h1 dir="auto">How does it work</h1>
<p dir="auto">We 've re-added the mypy pre-commit now - with mypy bumped to 0.910. This version detects far more errors and we should fix them all before we switch the CI check back.</p>
<ul dir="auto">
<li>
<p dir="auto">mypy will be running for incremental changes in pre-commit, same as before. This will enable incremental fixes of the code changed by committers who use pre-commits locally</p>
</li>
<li>
<p dir="auto">mypy on CI runs in non-failing mode. When the main pre-commit check is run, mypy is disabled, but then it is run as a separate step (which does not fail but will show the result of running mypy on all our code). This will enable us to track the progress of fixes</p>
</li>
</ul>
<h1 dir="auto">Can I help with the effort, you ask?</h1>
<p dir="auto">We started concerted effort now and incrementally fix all the mypy incompatibilities - ideally package/by/package to avoid huge code reviews. We'd really appreciate a number of people to contribute, so that we can re-enable mypy back fully and quickly :).</p>
<h1 dir="auto">How can I help?</h1>
<p dir="auto">What you need is:</p>
<ul dir="auto">
<li>checkout <code class="notranslate">main</code></li>
<li><code class="notranslate">./breeeze build-image</code></li>
<li><code class="notranslate">pip install pre-commit</code></li>
<li><code class="notranslate">pre-commit install</code></li>
</ul>
<p dir="auto">This will enable automated checks for when you do a regular contribution.  When you make your change, any MyPy issues will be reporteed and you need to fix them all to commit. You can also commit with <code class="notranslate">--no-verify</code> flag to skip that, bu, well, if you can improve airlfow a little - why not?</p>
<h1 dir="auto">How can I help more ?</h1>
<p dir="auto">You can add PRs that are fixing whole packages, without contributing features or bugfixes. Please refer to this issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1067158141" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/19891" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/19891/hovercard" href="https://github.com/apache/airflow/issues/19891">#19891</a> and ideally comment below in the issue that you want to take care of a package (to avoid duplicate work).</p>
<p dir="auto">An easy way to run MyPy check for package can be done either from the host:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="find DIRECTORY -name &quot;*.py&quot; | xargs pre-commit run mypy --files"><pre class="notranslate"><code class="notranslate">find DIRECTORY -name "*.py" | xargs pre-commit run mypy --files
</code></pre></div>
<p dir="auto">or from ./breeze shell:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="mypy --namespace-packages DIRECTORY"><pre class="notranslate"><code class="notranslate">mypy --namespace-packages DIRECTORY
</code></pre></div>
<h1 dir="auto">Current list of mypy PRs:</h1>
<p dir="auto"><a href="https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+label%3Amypy">https://github.com/apache/airflow/pulls?q=is%3Aopen+is%3Apr+label%3Amypy</a></p>
<h1 dir="auto">Remaining packages</h1>
<p dir="auto">Here is the list of remaining packages to be "mypy compliant" generated with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pre-commit run mypy --all-files 2&gt;&amp;1 | sed -r &quot;s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g&quot; | grep &quot;error:&quot; | sort | awk 'FS=&quot;:&quot; { print $1 }' | xargs dirname | sort | uniq -c | xargs -n 2 printf &quot;* [ ] (%4d) %s\n&quot;"><pre class="notranslate"><code class="notranslate">pre-commit run mypy --all-files 2&gt;&amp;1 | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" | grep "error:" | sort | awk 'FS=":" { print $1 }' | xargs dirname | sort | uniq -c | xargs -n 2 printf "* [ ] (%4d) %s\n"
</code></pre></div>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) airflow/api/common/experimental</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) airflow/contrib/sensors</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) airflow/example_dags</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) airflow/jobs</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   4) airflow/models</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) airflow/providers/microsoft/winrm/hooks</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) airflow/providers/ssh/hooks</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) tests/providers/amazon/aws/hooks</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> (   1) tests/providers/google/cloud/hooks</li>
</ul>
<h3 dir="auto">Committer</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I acknowledge that I am a maintainer/committer of the Apache Airflow project.</li>
</ul>
<p dir="auto"><strong>Description</strong><br>
Allow DAG to accept list of cron expression and schedule the dag in correlation to all of them.<br>
Similar to how it can be done in cron job</p>
<p dir="auto"><strong>Use case / motivation</strong><br>
Some scheduling like: every 10 min between 16:30 to 18:10 can not be obtained with single cron expression. The idea is that DAG will have the ability to be set according to more than 1 cron but without duplicating the DAG code or the DAG entry in the UI</p>
<p dir="auto">Even simple scheduling which is common for ETL : bi-weekly can not be done with single cron expression: <a href="https://serverfault.com/questions/404398/how-to-schedule-a-biweekly-cronjob" rel="nofollow">https://serverfault.com/questions/404398/how-to-schedule-a-biweekly-cronjob</a></p>