The Unbundling of Airflow
This post is hard to follow. But I'll give my unsolicited opinion on airflow:Its too complex to run as a single team and there are far better tools out there for scheduling. Airflow only makes sense when you need complex logic surrounding when to run jobs, how to backfill, when to backfill, and complex dependency trees. Otherwise, you are much better off with something like AWS step functions.