title: GCP Autoscale Down Protection
body: <p>So I have a set of long running tasks that have to be run on Compute Engine and have to scale. Each task takes approximately 3 hours. So in order to handle this I thought about using: </p>
  
  <p><a href="https://cloud.google.com/solutions/using-cloud-pub-sub-long-running-tasks" rel="nofollow noreferrer">https://cloud.google.com/solutions/using-cloud-pub-sub-long-running-tasks</a></p>
  
  <p>Architecture. And while it works fine there is one huge problem. On scale down, I'd really like to avoid it scaling down a task that is currently running! I'd potentially lose 3 hours worth of processing. </p>
  
  <p>Is there a way to ensure that autoscale down doesn't scale down a VM with a long running / uptime? </p>
  
  <p>EDIT: A few people have asked to elaborate my task. So it's similar to what's described in the link above which is many long running tasks that need to be run on a GPU. There is a chunk of data that needs to be processed. It takes 4 hours (video encoding) then once completed it outputs to a bucket. Well it can take anywhere from 1 to 6 hours depending on the length of the video. Just like the architecture above it would be nice to have the cluster scale up based on queue size. But when scaling down I'd like to ensure that it's not scaling down currently running tasks which is what is currently happening. It being GPU bound doesn't allow me to use the CPU metric.  </p>