The short version: any complex data processing done at scale will have a lot of steps and fail a lot. To avoid having to constantly monitor everything, you have systems called &quot;workflow managers&quot; that help you make sense of those multi steps monsters and manage their dependencies, their failure etc...<p>If you never had to process that kind of data (we&#x27;re talking about Google, Facebook and Twitter kind of data) then you probably have no exposure to this kind of system.