Data Enginnering, Data Pipeline, Etl, Workflow Management, Airflow.
notion and upstream pipeline signal In this technique the downstream pipelines will still be scheduled using Technique #1 but only execute when the upstream pipeline has signaled it has successfully completed its current run. In other words, the downstream pipelines wait for some sort of a signal