<p dir="auto"><strong>Description</strong></p>
<p dir="auto">The old documentation for <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=63407290" rel="nofollow">Scheduler Basics</a> should be reviewed and incorporated into <a href="https://github.com/apache/airflow/tree/master/docs">Airflow Documentation</a></p>
<p dir="auto"><strong>Use case / motivation</strong></p>
<p dir="auto">To remove documentation duplication and clean it up.</p>
<p dir="auto"><strong>Related Issues</strong></p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="673709470" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/10178" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/10178/hovercard" href="https://github.com/apache/airflow/issues/10178">#10178</a></p>
<h3 dir="auto">Apache Airflow Provider(s)</h3>
<p dir="auto">google</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto">apache-airflow 2.5.0<br>
apache-airflow-providers-apache-beam 4.1.0<br>
apache-airflow-providers-cncf-kubernetes 5.0.0<br>
apache-airflow-providers-google 8.6.0<br>
apache-airflow-providers-grpc 3.1.0</p>
<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.5.0</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Debian 11</p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Official Apache Airflow Helm Chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">KubernetesExecutor</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">GCSToBigQueryOperator allows multiple ways to specify schema of the BigQuery table:</p>
<ol dir="auto">
<li>Setting autodetect == True</li>
<li>Setting schema_fields directly with autodetect == False</li>
<li>Setting a schema_object and optionally a schema_object_bucket with autodetect == False</li>
</ol>
<p dir="auto">This third method seems to be broken in the latest provider version (8.6.0) and will always result in this error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2022-12-16, 21:06:18 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
  File &quot;/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py&quot;, line 395, in execute
    self.configuration = self._check_schema_fields(self.configuration)
  File &quot;/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py&quot;, line 524, in _check_schema_fields
    raise RuntimeError(
RuntimeError: Table schema was not found. Set autodetect=True to automatically set schema fields from source objects or pass schema_fields explicitly"><pre class="notranslate"><code class="notranslate">[2022-12-16, 21:06:18 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py", line 395, in execute
    self.configuration = self._check_schema_fields(self.configuration)
  File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py", line 524, in _check_schema_fields
    raise RuntimeError(
RuntimeError: Table schema was not found. Set autodetect=True to automatically set schema fields from source objects or pass schema_fields explicitly
</code></pre></div>
<p dir="auto">The reason for this is because <a href="https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L318-L320">this block</a> where <code class="notranslate">if self.schema_object and self.source_format != "DATASTORE_BACKUP":</code>. fails to set self.schema_fields. It only sets the local variable, schema_fields. When self._check_schema_fields is subsequently called <a href="https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L395">here</a>, we enter the <a href="https://github.com/apache/airflow/blob/25bdbc8e6768712bad6043618242eec9c6632618/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py#L523-L528">first block</a> because autodetect is false and schema_fields is not set.</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">No error should be raised if autodetect is set to False and a valid schema_object is provided</p>
<h3 dir="auto">How to reproduce</h3>
<ol dir="auto">
<li>Create a simple BigQuery table with a single column col1:</li>
</ol>
<div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE `my-project.my_dataset.test_gcs_to_bigquery` (col1 INT);"><pre class="notranslate">CREATE TABLE <span class="pl-s"><span class="pl-pds">`</span>my-project.my_dataset.test_gcs_to_bigquery<span class="pl-pds">`</span></span> (col1 <span class="pl-k">INT</span>);</pre></div>
<ol start="2" dir="auto">
<li>Upload a json blob for this object to a bucket (e.g., data/schemas/table.json)</li>
<li>Upload a simple CSV for the source file to load to a bucket (e.g., data/source/file.csv)</li>
<li>Run the following command:</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="    gcs_to_biquery = GCSToBigQueryOperator(
        task_id=&quot;gcs_to_bigquery&quot;,
        destination_project_dataset_table=&quot;my-project.my_dataset.test_gcs_to_bigquery&quot;,
        bucket=&quot;my_bucket_name&quot;,
        create_disposition=&quot;CREATE_IF_NEEDED&quot;,
        write_disposition=&quot;WRITE_TRUNCATE&quot;,
        source_objects=[&quot;data/source/file.csv&quot;],
        source_format=&quot;CSV&quot;,
        autodetect=False,
        schema_object=&quot;data/schemas/table.json&quot;,
    )"><pre class="notranslate">    <span class="pl-s1">gcs_to_biquery</span> <span class="pl-c1">=</span> <span class="pl-v">GCSToBigQueryOperator</span>(
        <span class="pl-s1">task_id</span><span class="pl-c1">=</span><span class="pl-s">"gcs_to_bigquery"</span>,
        <span class="pl-s1">destination_project_dataset_table</span><span class="pl-c1">=</span><span class="pl-s">"my-project.my_dataset.test_gcs_to_bigquery"</span>,
        <span class="pl-s1">bucket</span><span class="pl-c1">=</span><span class="pl-s">"my_bucket_name"</span>,
        <span class="pl-s1">create_disposition</span><span class="pl-c1">=</span><span class="pl-s">"CREATE_IF_NEEDED"</span>,
        <span class="pl-s1">write_disposition</span><span class="pl-c1">=</span><span class="pl-s">"WRITE_TRUNCATE"</span>,
        <span class="pl-s1">source_objects</span><span class="pl-c1">=</span>[<span class="pl-s">"data/source/file.csv"</span>],
        <span class="pl-s1">source_format</span><span class="pl-c1">=</span><span class="pl-s">"CSV"</span>,
        <span class="pl-s1">autodetect</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
        <span class="pl-s1">schema_object</span><span class="pl-c1">=</span><span class="pl-s">"data/schemas/table.json"</span>,
    )</pre></div>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>