Pinterest, Engineering, Infrastructure, Open Source.
Spark. ML engineers chain a sequence of Spark and Pytorch jobs using Airflow, and package them as “workflow templates” that can be reused to produce new model training DAGs quickly. However, as ML is rapidly evolving, not all dataset iteration needs can be supported quickly by workflow templates.