I’m using the GoogleCloudStorageToBigQueryOperator to create a small table in BigQuery using a csv file in GCS. I have the airflow in a VirtualBox on my local machine. Every time, this simple operation takes 15 minutes exactly to complete. I’ve tried changing from Local to Celery Executor but it still takes 15 minutes. Any suggestions please to improve the performance?
Thanks a lot,
Sri.
I've created a VM on the GCP and installed airflow on it. Moved the DAG on to it and the data got loaded within seconds.
Airflow - GCP Operator: GoogleCloudStorageToBigQueryOperator - Takes 15 minutes to complete