<p dir="auto"><strong>Apache Airflow version</strong>: None given</p>
<p dir="auto">Ticket was created 21/Nov/16 18:35</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>:</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):</li>
<li><strong>Install tools</strong>:</li>
<li><strong>Others</strong>:<br>
<strong>What happened</strong>:</li>
</ul>
<p dir="auto">We are planning to run around 40,000 tasks a day using airflow and some of them are critical to give quick feedback to developers.</p>
<p dir="auto">Currently having execution date to uniquely identify tasks does not work for us since we mainly trigger dags (instead of running them on schedule) we collide with 1 sec granularity on several occasions.  Having a task uuid or associating dag_run to task_instance  table and using this for scheduling and updating status will help us here.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<p dir="auto"><strong>Anything else we need to know</strong>:</p>
<p dir="auto">Moved here from <a href="https://issues.apache.org/jira/browse/AIRFLOW-642" rel="nofollow">https://issues.apache.org/jira/browse/AIRFLOW-642</a></p>
<p dir="auto"><strong>Apache Airflow version</strong>: 1.10.*<br>
<strong>Backport packages version</strong>: 2020.10.5rc1</p>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Although good work has been done to improve the <code class="notranslate">BigQueryInsertJobOperator</code> by the assigning of a <code class="notranslate">job_id</code> based on a combination of <code class="notranslate">dag_id</code>, <code class="notranslate">task_id</code>, <code class="notranslate">execution_date</code> and an additional <code class="notranslate">uniqueness_suffix</code>, other operators that create BigQuery jobs - e.g. <code class="notranslate">GCSToBigQueryOperator </code> - do not take advantage of this but instead still create a job_id based on a timestamp.</p>
<p dir="auto"><a href="https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/bigquery.py#L1475">https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/bigquery.py#L1475</a></p>
<p dir="auto">If more than one task based on this operator are launched at the same time, this will cause an error due to a duplication in the job_id.</p>
<p dir="auto"><code class="notranslate">Already Exists: Job my-project:EU.airflow_1601894411</code></p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
Any task that starts a BigQuery should create a job_id that is unique.</p>
<p dir="auto">Perhaps the method of <code class="notranslate">job_id</code> creation needs to move into the BigQuery hook, if possible?<br>
<a href="https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/bigquery.py#L2060">https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/bigquery.py#L2060</a></p>