<h3 dir="auto">What do you see as an issue?</h3>
<p dir="auto">The <a href="https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html" rel="nofollow">documentation</a> for the template variable <code class="notranslate">data_interval_end</code> states simply that its the "End of the data interval". Initially I took that to mean the final second/microsecond (depending on precision) of the data interval, but its actually the start of the next interval.</p>
<p dir="auto">For example,  given a schedule that runs every 5 minutes, the variables are set as follows (for a simple test I just ran):</p>
<ul dir="auto">
<li><strong>data_interval_start</strong>: 2022-07-28T16:15:00+00:00</li>
<li><strong>data_interval_end</strong>: 2022-07-28T16:20:00+00:00</li>
</ul>
<p dir="auto">As opposed to:</p>
<ul dir="auto">
<li><strong>data_interval_start</strong>: 2022-07-28T16:15:00+00:00</li>
<li><strong>data_interval_end</strong>: 2022-07-28T16:19:59+00:00</li>
</ul>
<p dir="auto">It makes sense as implemented, but I'd like to see the documentation state the behavior explicitly.</p>
<h3 dir="auto">Solving the problem</h3>
<p dir="auto">The documentation I read when using the template variables is <a href="https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html" rel="nofollow">https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html</a></p>
<p dir="auto">I'm not sure how to word it, but it would be helpful to indicate that <code class="notranslate">data_interval_end</code> is effectively the <code class="notranslate">data_interval_start</code> of the next interval. Or that the interval the DAG is operating on, using mathematical interval notation, is <code class="notranslate">[data_interval_start, data_interval_end)</code></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto">Just an explanation of how we ran into this:</p>
<p dir="auto">We are running DAGS periodically that pull timeseries data from some API. The jobs query an api with filters like this:</p>
<ul dir="auto">
<li>timestamp &gt;= data_interval_start</li>
<li>timestamp &lt;= data_interval_end</li>
</ul>
<p dir="auto">We noticed we were getting some duplicated data, specifically data that happened exactly on the <code class="notranslate">data_interval_start</code> value. The simple fix (once we saw the behavior of the variables) was to remove the <code class="notranslate">=</code> from the second filter:</p>
<ul dir="auto">
<li>timestamp &gt;= data_interval_start</li>
<li>timestamp &lt; data_interval_end</li>
</ul>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<h3 dir="auto">Description</h3>
<p dir="auto">In our use case, we often import data which contains some noise which cannot be handled correctly. However, we don't want an entire task or job to fail because of it (simply because it is not critical). A warning shall be logged to the corresponding data point and the job shall carry on as usual.</p>
<p dir="auto"><strong>The problem</strong>: a mere <code class="notranslate">logging.warning()</code> stays in the hiding when nobody looks at the logs (why should they when everything succeeded?). Also, having a separate task/job which checks for those kind of errors is adding too much complexity to the logic and makes it hard to sync to the origin.</p>
<p dir="auto"><strong>The proposal</strong>: Add a new task return status <code class="notranslate">Success with warning</code> or simply <code class="notranslate">Warning</code> which shows at high level, what's going on in the tasks. Users can see that there is some action needed, but not immediately, without having to read each of the logs or having to setup additional monitoring levels.</p>
<p dir="auto">If someone points me toward the relevant parts in the application logic that should be modified, I would be happy to have a shot at it myself.</p>
<h3 dir="auto">Use case/motivation</h3>
<p dir="auto">Warnings in tasks should be made visible in the UI at job level (through a status flag).</p>
<h3 dir="auto">Related issues</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit a PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>