Answers.Body: <p>Cromwell will send jobs to the AWS Batch queue ARN defined in the <code>cromwell.conf</code> file in the Cromwell server. To use Spot instances you can create an AWS Batch Compute environment configured to use Spot instances and create a queue for that compute environment. You can then set the arn of that queue in the configuration file or override the configured value using the standard mechanisms of overriding Cromwell’s configuration (workflow options, runtime parameters etc). If you change the configuration file you will need to restart the server.</p>
  <p>Batch queues can point to more than one compute environment. Each will have a priority. Jobs in a queue will be sent to the highest priority environment until that environment has reached its max vCPU capacity, at that point jobs will be sent to the next environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T17:24:27.033Z
Answers.LastActivityDate: 2021-08-28T17:29:53.207Z
Answers.LastEditDate: 2021-08-28T17:29:53.207Z
Answers.UserDisplayName: Mark
Body: <p>For processing genomics workflows I make use of widdle and cromwell with an AWS back-end. I have followed all examples on <a href="https://docs.opendata.aws/genomics-workflows/orchestration/cromwell/cromwell-overview/" rel="nofollow noreferrer">the AWS cromwell tutorial</a>. In my AWS batch console I can see there are 2 job queues (priority-cromwell and default-cromwell) and both queues can submit jobs to 2 compute environments (ondemand-cromwell and spot-cromwell). I have however no idea how I can control which jobs are put in which queue and are executed in which compute environment. I'd like to make optimal use of the spot-instances, how can I do that? Do I have to add something in the runtime block of my widdle task definitions, should I include something in the cromwell configuration file, or are for example only scattered jobs executed on spot instances?</p>
  
Tags: <workflow-management><aws>
Title: How does cromwell and AWS batch decide which compute environment (on-demand or spot) to use?
ThreadText: # How does cromwell and AWS batch decide which compute environment (on-demand or spot) to use?
  
  (Asked by: username_0 on 2020-11-13)
  
  For processing genomics workflows I make use of widdle and cromwell with an AWS back-end. I have followed all examples on [the AWS cromwell tutorial](https://docs.opendata.aws/genomics-workflows/orchestration/cromwell/cromwell-overview/). In my AWS batch console I can see there are 2 job queues (priority-cromwell and default-cromwell) and both queues can submit jobs to 2 compute environments (ondemand-cromwell and spot-cromwell). I have however no idea how I can control which jobs are put in which queue and are executed in which compute environment. I'd like to make optimal use of the spot-instances, how can I do that? Do I have to add something in the runtime block of my widdle task definitions, should I include something in the cromwell configuration file, or are for example only scattered jobs executed on spot instances?
  
  **Tags:** workflow-management, aws
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-28. Score: 1
  
  Cromwell will send jobs to the AWS Batch queue ARN defined in the `cromwell.conf` file in the Cromwell server. To use Spot instances you can create an AWS Batch Compute environment configured to use Spot instances and create a queue for that compute environment. You can then set the arn of that queue in the configuration file or override the configured value using the standard mechanisms of overriding Cromwell’s configuration (workflow options, runtime parameters etc). If you change the configuration file you will need to restart the server.
  
  Batch queues can point to more than one compute environment. Each will have a priority. Jobs in a queue will be sent to the highest priority environment until that environment has reached its max vCPU capacity, at that point jobs will be sent to the next environment.
  
  
  ---
  
  