Assuming you&#x27;re running multiple pipelines at once you can add robustness in this sort of scenario by running the long-running tasks on guaranteed VMs (complete with live migration on GCE) and the short-running tasks on cheap preemptible VMs.<p>Another features that comes to mind for enabling this sort of work is GCE&#x27;s support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations (&quot;hey, this directory contains the complete results of stage &lt;x&gt;, one of you stage &lt;y&gt; workers can pick it up&quot; or &quot;I have completed processing the stage &lt;x&gt; outputs for job &lt;5&gt;; they can be removed or archived to GCS.&quot;). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume).<p>Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they&#x27;ll handle the work balancing for you.<p>(note: I work on GCE)