<h3 dir="auto">Description</h3>
<p dir="auto">One of the main things we want to know is task durations.</p>
<p dir="auto">The main way I know how to get this is to hover over the <code class="notranslate">Gannt</code> chart or use the <code class="notranslate">Task Duration</code> chart.</p>
<p dir="auto">It would be nice to have the data that is in the <code class="notranslate">Gannt</code> chart or the <code class="notranslate">Task Duration</code> chat in a table view.</p>
<h3 dir="auto">Use case/motivation</h3>
<p dir="auto">That plots are making the data harder to find than a table alone.</p>
<h3 dir="auto">Related issues</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit a PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<h3 dir="auto">Describe the issue with documentation</h3>
<p dir="auto">In response to the contents of <a href="https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#testing" rel="nofollow">https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#testing</a> as of 2021-10-13</p>
<p dir="auto">There are several broken things in the tutorial as given, and a few pieces of missing context. (I can help PR this once I go through the contributor guidelines; just recording them for now).</p>
<p dir="auto">In order:</p>
<h2 dir="auto">Table Definitions</h2>
<p dir="auto">In <a href="https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#initial-setup" rel="nofollow">initial setup</a>, there is a name collision in the <code class="notranslate">employees_temp</code>: both tables use a constraint named <code class="notranslate">employees_pk</code>. The latter (<code class="notranslate">_temp</code>) should probably use <code class="notranslate">employees_temp_pk</code>.</p>
<p dir="auto">(As a nit-pick, it feels odd to capitalize the table names, requires quoting the names when using <code class="notranslate">psql</code>'s <code class="notranslate">\d</code>.)</p>
<h2 dir="auto">Python Code</h2>
<h3 dir="auto">Imports</h3>
<p dir="auto">Firstly, it's missing necessary imports. Perhaps they're obvious if you're experienced with airflow, but for those coming into fresh it's not clear what's missing. It appears to be:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from airflow.decorators import dag, task
from airflow.hooks.postgres_hook import PostgresHook
from datetime import datetime, timedelta
import requests"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">decorators</span> <span class="pl-k">import</span> <span class="pl-s1">dag</span>, <span class="pl-s1">task</span>
<span class="pl-k">from</span> <span class="pl-s1">airflow</span>.<span class="pl-s1">hooks</span>.<span class="pl-s1">postgres_hook</span> <span class="pl-k">import</span> <span class="pl-v">PostgresHook</span>
<span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">datetime</span>, <span class="pl-s1">timedelta</span>
<span class="pl-k">import</span> <span class="pl-s1">requests</span></pre></div>
<h3 dir="auto">Connections</h3>
<p dir="auto">There's no mention of 'connections' (or the concept) in the documentation leading up to this tutorial. The code as written relies on an airflow <code class="notranslate">Connection</code> named <code class="notranslate">LOCAL</code> being defined.</p>
<h3 dir="auto">Paths and Duplicated Code</h3>
<p dir="auto">The path where the file is downloaded is hard-coded to <code class="notranslate">/usr/local/airflow/...</code>. For those following <a href="https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html" rel="nofollow">https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html</a> this would likely need to be <code class="notranslate">/opt/airflow/dags/files</code>, and require the <code class="notranslate">files</code> subdirectory to be created first.<br>
Regardless, it would be nice if this path were defined once at the top of the script or as a parameter or other config option rather than copy-pasted.</p>
<h3 dir="auto">Bugs</h3>
<p dir="auto"><code class="notranslate">cur.copy_from</code> references file handle with variable <code class="notranslate">f</code> but the handle's name is <code class="notranslate">file</code>.</p>
<p dir="auto"><code class="notranslate">copy_from</code> does not handle a CSV header row. We either need to skip the first line, or use <code class="notranslate">copy_expert</code> or similar:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="cur.copy_expert(&quot;COPY \&quot;Employees_temp\&quot; FROM stdin WITH CSV HEADER DELIMITER AS ','&quot;, file)"><pre class="notranslate"><span class="pl-s1">cur</span>.<span class="pl-en">copy_expert</span>(<span class="pl-s">"COPY <span class="pl-cce">\"</span>Employees_temp<span class="pl-cce">\"</span> FROM stdin WITH CSV HEADER DELIMITER AS ','"</span>, <span class="pl-s1">file</span>)</pre></div>
<h3 dir="auto">Location</h3>
<p dir="auto">It says 'save the file' but doesn't specify whether there's any naming convention or somesuch. I might suggest providing an example name like <code class="notranslate">etl.py</code>.</p>
<h2 dir="auto">Data</h2>
<p dir="auto">The data this script relies on comes from <a href="https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/pipeline_example.csv" rel="nofollow">https://raw.githubusercontent.com/apache/airflow/main/docs/apache-airflow/pipeline_example.csv</a> .</p>
<p dir="auto">It appears this file is a spreadsheet export; the 'Serial Number<code class="notranslate">column's data is in scientific notation format. This is resulting in some truncation and duplicate keys (e.g.</code>9.78938E+12` appears 25 times).</p>
<p dir="auto">Secondly, the data at this URL appears to use linefeed (LF,<code class="notranslate">\n</code>) for newlines. The postgres 'copy' documentation says this should work fine.<br>
The python code itself <em>before</em> attempting the <code class="notranslate">copy_from</code> splits the input on newline; effectively <em>removing</em> all newlines from the file.  Seems we should remove the 'split' and just write the contents directly, or have it <code class="notranslate">file.write(row+'\n')</code></p>
<p dir="auto">Lastly, there's a blank newline at the end of the written file; this doesn't work for postgres <code class="notranslate">copy from</code></p>
<h3 dir="auto">How to solve the problem</h3>
<p dir="auto">Just to re-itemize from the above:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked="">  include all necessary imports in the example</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> move the saved file path to a variable or config with notes to modify it to meet the current system's requirements</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> include a note on <code class="notranslate">Connections</code> and note that one needs to be set up as appropriate (either one named <code class="notranslate">LOCAL</code> or the user should create one and modify the code here to use it). This may be a bigger concern for those following the [docker start](<a href="https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html" rel="nofollow">https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> avoid duplicate primary key constraint names in the table definitions</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> retain newlines when writing CSV file and remove blank lines</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> correct variable names (<code class="notranslate">file</code> instead of <code class="notranslate">f</code>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> properly handle CSV header row</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> use <code class="notranslate">long</code> number format rather than scientific notation</li>
</ul>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>