<p dir="auto">Hello,<br>
I'm running Airflow 1.10.10 and using the <code class="notranslate">GCSToBigQueryOperator</code> from <code class="notranslate">airflow.providers.google.cloud.transfers.gcs_to_bigquery</code> in order to load data from google cloud storage into a big query table.<br>
Although everything is working as expected, I keep seeing (a lot of) deprecation warnings in my logs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DeprecationWarning: The bigquery_conn_id parameter has been deprecated. You should pass the gcp_conn_id parameter.
bigquery.py:103: DeprecationWarning: This method will be deprecated. Please use `BigQueryHook.get_client` method`
taskinstance.py:984: DeprecationWarning: This method is deprecated. Please use airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_load`
bigquery.py:1614: DeprecationWarning: This method is deprecated. Please use `BigQueryHook.insert_job` method."><pre class="notranslate"><code class="notranslate">DeprecationWarning: The bigquery_conn_id parameter has been deprecated. You should pass the gcp_conn_id parameter.
bigquery.py:103: DeprecationWarning: This method will be deprecated. Please use `BigQueryHook.get_client` method`
taskinstance.py:984: DeprecationWarning: This method is deprecated. Please use airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_load`
bigquery.py:1614: DeprecationWarning: This method is deprecated. Please use `BigQueryHook.insert_job` method.
</code></pre></div>
<p dir="auto">Although everything is working well, I'm afraid that in the future those warnings will indeed cause problems.<br>
Please update the code and find a way to use the suggested methods instead of the deprecated ones.<br>
If any more information is needed from me regarding the warnings or my system I will be happy to add it.<br>
Thank you!</p>
<p dir="auto">Current issue is I have pools configured on same type of task where I can have up to max 100 pool slots of that task type running in parallel BUT I want ability to say don't START running more than 5 at a time. So ideally these tasks should be scheduled in steps.<br>
ie<br>
at each scheduler iteration 1 (ie 9:03:30 am) 5 tasks scheduled for the pool<br>
at each scheduler iteration 2 (ie 9:04:00 am) 5 more tasks scheduled for the pool (so 10 currently running in the pool)<br>
at each scheduler iteration 3 (ie 9:04:30 am) 5 more tasks scheduled for the pool (so 15 currently running in the pool)<br>
at 9:04:42 am 3 tasks completed for the pool<br>
at each scheduler iteration 4 (ie 9:05:00 am) 5 more tasks scheduled for the pool (so 17 currently running in the pool)<br>
..etc</p>