Lessons learned from running Apache Airflow at scale
Unless you have extremely complex dependency graphs, I really don't think airflow is worth it. It's very easy to end up essentially writing an "orchestrator" using airflow, it allows for very flexible low level operations. The added complexity has minimal benefit, and like something like apache spark, what looks simple becomes hard to reason about in real world scenarios. You need to understand how it works under the hood, and get the best practices right.As mentioned elsewhere, AWS step functions are really the best in orchestration.