Answers.Body: <p>You didn't say what version of the AWS CLI you're using, but I suspect it's version 1. It looks like the <code>evaluateOnExit</code> feature was added in version 2 but was not backported. Installing AWS CLI version 2 resolves the issue:</p>
  <p><a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html</a></p>
  <p>Note that <a href="https://www.nextflow.io/" rel="nofollow noreferrer">Nextflow</a> also provides built-in support for <a href="https://www.nextflow.io/docs/latest/aws.html#aws-batch" rel="nofollow noreferrer">AWS Batch</a>. You'd still need to configure some AWS infrastructure using Nextflow, but it should be a lot less painful than writing/maintaining your own script(s). It's definitely worth checking out if you haven't already.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-31T01:09:41.197Z
Answers.LastActivityDate: 2022-10-31T01:09:41.197Z
Answers.UserDisplayName: Steve
Body: <p>I am in the process of creating a bioinformatics pipeline and I am currently trying to setup the AWS infrastructure for which it will run.</p>
  <p>The script I am running to setup the AWS batch job looks like the following:</p>
  <pre><code>#! /bin/bash
  
  # Setup the AWS Batch architecture
  # Copyright 2022, SolareaBio
  
  set -e
  
  
  # ***** BEGIN custom lines
  
  container=&quot;BGC_bacteria&quot;
  vCPUs_req=12
  memory_MiB_req=40000
  # ASK ABOUT CUSTOM CPU AND MEMORY SETTINGS
  
  # ***** END custom lines
  
  
  
  container_lc=<span class="math-container">$(echo $</span>container | tr [A-Z] [a-z])
  
  subnet_useast1a=&quot;subnet-42371d0a&quot;
  subnet_useast1b=&quot;subnet-b03c54ea&quot;
  subnet_useast1c=&quot;subnet-00ee1364&quot;
  
  usage() {
    cat &lt;&lt; EOF
  Usage $0 [-h]
  
  Builds the <span class="math-container">$container docker container, pushes it to the AWS ECR, and sets up the $</span>container AWS Batch environment
  EOF
  }
  
  while getopts &quot;h&quot; option; do
    case $option in
      h)   usage;exit;;
      [?]) usage; exit;;
    esac
  done
  
  script_dir=<span class="math-container">$(dirname "$</span>0&quot;)
  
  repo=&quot;990850896279.dkr.ecr.us-east-1.amazonaws.com&quot;
  
  echo &quot;Logging into ECR&quot;
  aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${repo}
  
  image_and_tag=&quot;<span class="math-container">${container_lc}:latest"
  echo "Building the docker container: $</span>image_and_tag&quot;
  
  # If you get errors from the docker build command, try adding --no-cache
  docker build --no-cache -f <span class="math-container">$script_dir/$</span>{container}.dockerfile -t <span class="math-container">$image_and_tag $</span>script_dir
  
  echo &quot;Tagging Image&quot;
  docker tag <span class="math-container">${image_and_tag} $</span>{repo}/${image_and_tag}
  
  # Create repository in AWS ECR
  repo_container_info=<span class="math-container">$(aws ecr describe-repositories --repository-names $</span>{container_lc})
  if [[ &quot;<span class="math-container">$repo_container_info" == "" ]]; then
    aws ecr create-repository --repository-name $</span>{container_lc}
  fi
  
  echo &quot;Pushing Image up to AWS&quot;
  docker push <span class="math-container">${repo}/$</span>{image_and_tag}
  
  # Create compute environment
  EC2_type=ON_DEMAND
  ce_info=<span class="math-container">$(aws batch describe-compute-environments --compute-environments $</span>{container}_<span class="math-container">${EC2_type}_CE)
  ce_status=$</span>(echo <span class="math-container">$ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
  if [[ $</span>ce_status != &quot;VALID&quot; ]]; then
    compute_resources='{&quot;type&quot;:&quot;EC2&quot;,&quot;allocationStrategy&quot;:&quot;BEST_FIT_PROGRESSIVE&quot;,&quot;minvCpus&quot;:0,&quot;maxvCpus&quot;:10000,&quot;tags&quot;:{&quot;Name&quot;:&quot;'<span class="math-container">${container}_Batch'"},"instanceTypes":["c5a","m5a","c5","m5"],"subnets":["'$</span>subnet_useast1a'&quot;,&quot;'<span class="math-container">$subnet_useast1b'","'$</span>subnet_useast1c'&quot;],&quot;securityGroupIds&quot;:[&quot;sg-ff2fd48f&quot;,&quot;sg-65ff5315&quot;],&quot;ec2KeyPair&quot;:&quot;SBTools&quot;,&quot;instanceRole&quot;:&quot;ecsInstanceRole&quot;}'
    aws batch create-compute-environment \
      --compute-environment-name <span class="math-container">${container}_$</span>{EC2_type}_CE \
      --type MANAGED \
      --state ENABLED \
      --compute-resources $compute_resources \
      --service-role AWSBatchServiceRole
  
    printf &quot;Waiting for Compute Environment to be created&quot;
    while [[ <span class="math-container">$ce_status != "VALID" ]]; do
  sleep 5
  printf "."
  ce_info=$</span>(aws batch describe-compute-environments --compute-environments <span class="math-container">${container}_$</span>{EC2_type}_CE)
      ce_status=<span class="math-container">$(echo $</span>ce_info | jq '.computeEnvironments[0] | .status // &quot;&quot;' | tr -d '&quot;')
    done
    echo &quot;&quot;
  fi
  
  # Create the job queue
  jq_info=<span class="math-container">$(aws batch describe-job-queues --job-queues $</span>{container}_<span class="math-container">${EC2_type}_JQ)
  jq_status=$</span>(echo <span class="math-container">$jq_info | jq '.jobQueues[0] | .status // ""' | tr -d '"')
  if [[ $</span>jq_status != &quot;VALID&quot; ]]; then
    aws batch create-job-queue \
      --job-queue-name <span class="math-container">${container}_$</span>{EC2_type}_JQ \
      --state ENABLED \
      --priority 5 \
      --compute-environment-order '{&quot;order&quot;:1,&quot;computeEnvironment&quot;:&quot;'<span class="math-container">${container}_$</span>{EC2_type}_CE'&quot;}'
  fi
  
  # Register job definition
  job_definition=&quot;<span class="math-container">${container}_JD"
  jd_info=$</span>(aws batch describe-job-definitions --status ACTIVE --job-definition-name <span class="math-container">$job_definition)
  jd_status=$</span>(echo <span class="math-container">$jd_info | jq '.jobDefinitions[0] | .status // ""' | tr -d '"')
  if [[ $</span>jd_status != &quot;ACTIVE&quot; ]]; then
    aws batch register-job-definition \
      --job-definition-name <span class="math-container">$job_definition \
      --type container \
      --container-properties '{"privileged":true,"image":"'$repo/$image_and_tag'","vcpus":'${vCPUs_req}',"memory":'$</span>{memory_MiB_req}',&quot;mountPoints&quot;:[{&quot;sourceVolume&quot;:&quot;SB-EFS&quot;,&quot;containerPath&quot;:&quot;/EFS&quot;}],&quot;volumes&quot;:[{&quot;host&quot;:{&quot;sourcePath&quot;:&quot;/&quot;},&quot;name&quot;:&quot;SB-EFS&quot;}],&quot;command&quot;:[&quot;Ref::container&quot;,&quot;Ref::run_id&quot;,&quot;Ref::input_file&quot;]}' \
      --retry-strategy '{&quot;attempts&quot;: 10,&quot;evaluateOnExit&quot;:[{&quot;onStatusReason&quot; :&quot;Host EC2*&quot;,&quot;action&quot;: &quot;RETRY&quot;},{&quot;onReason&quot; : &quot;*&quot;,&quot;action&quot;: &quot;EXIT&quot;}]}'
  fi
  
  # Make sure the EFS has an access point set up
  efs_id=<span class="math-container">$(aws efs describe-file-systems | jq -r '.FileSystems[] | select (.Name=="SB-EFS") | .FileSystemId')
  if [[ "$</span>efs_id&quot; == &quot;&quot; ]]; then
    echo &quot;ERROR: SB-EFS does not exist&quot;
    exit
  fi
  ap_id=<span class="math-container">$(aws efs describe-access-points | jq -r '.AccessPoints[] | select (.Name=="EFS_AP") | .AccessPointId')
  if [[ "$</span>ap_id&quot; == &quot;&quot; ]]; then
    # 1001:1001 is user sb in group sb
    ap_id=<span class="math-container">$(aws efs create-access-point --file-system-id "$</span>efs_id&quot; --posix-user '{&quot;Uid&quot;:1001,&quot;Gid&quot;:1001}' --root-directory '{&quot;Path&quot;:&quot;/&quot;}' --tags '[{&quot;Key&quot;:&quot;Name&quot;,&quot;Value&quot;:&quot;EFS_AP&quot;}]' | jq -r '.AccessPointId')
  fi  
  </code></pre>
  <p>Now when I run it, everything works as it should until getting to the --retry-strategy command line in the &quot;register job definition&quot; section of the script. The following error is subsequently thrown:</p>
  <pre><code>Parameter validation failed:
  Unknown parameter in retryStrategy: &quot;evaluateOnExit&quot;, must be one of: attempts
  </code></pre>
  <p>Not sure what the issue is with the &quot;attempts&quot; parameter that is set. Any potential solutions or guidance to solving this error is much appreciated. Thanks!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-28T21:08:57.347Z
Comments.Text: Hi @Joe_G, could you please provide a bit more information about what you're trying to do with this script? Stories help a *lot* with getting good, helpful answers. I can see something suggesting metagenomics, but it's not clear to me what a correct script would do.
Comments.UserDisplayName: gringer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-30T00:00:36.100Z
Comments.Text: My personal approach would be to use the NextFlow AWS pipeline. It would simplify everything, e.g. NextFlow automatically 'Dockerises'
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-19T19:13:30.047Z
Comments.Text: Just a suggestion, but in my experience it's much more readable to use a Python library for such things, try out boto3. I've never tried submitting batch jobs, but I believe it could help in debugging the issue !
Comments.UserDisplayName: h3ab74
Tags: <bash><shell><aws>
Title: Error Registering Job Definition AWS
ThreadText: # Error Registering Job Definition AWS
  
  (Asked by: username_0 on 2022-10-27)
  
  I am in the process of creating a bioinformatics pipeline and I am currently trying to setup the AWS infrastructure for which it will run.
  
  The script I am running to setup the AWS batch job looks like the following:
  
  ```
  #! /bin/bash
  
  # Setup the AWS Batch architecture
  # Copyright 2022, SolareaBio
  
  set -e
  
  # ***** BEGIN custom lines
  
  container="BGC_bacteria"
  vCPUs_req=12
  memory_MiB_req=40000
  # ASK ABOUT CUSTOM CPU AND MEMORY SETTINGS
  
  # ***** END custom lines
  
  container_lc=$(echo $container | tr [A-Z] [a-z])
  
  subnet_useast1a="subnet-42371d0a"
  subnet_useast1b="subnet-b03c54ea"
  subnet_useast1c="subnet-00ee1364"
  
  usage() {
    cat << EOF
  Usage $0 [-h]
  
  Builds the $container docker container, pushes it to the AWS ECR, and sets up the $container AWS Batch environment
  EOF
  }
  
  while getopts "h" option; do
    case $option in
      h)   usage;exit;;
      [?]) usage; exit;;
    esac
  done
  
  script_dir=$(dirname "$0")
  
  repo="990850896279.dkr.ecr.us-east-1.amazonaws.com"
  
  echo "Logging into ECR"
  aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${repo}
  
  image_and_tag="${container_lc}:latest"
  echo "Building the docker container: $image_and_tag"
  
  # If you get errors from the docker build command, try adding --no-cache
  docker build --no-cache -f $script_dir/${container}.dockerfile -t $image_and_tag $script_dir
  
  echo "Tagging Image"
  docker tag ${image_and_tag} ${repo}/${image_and_tag}
  
  # Create repository in AWS ECR
  repo_container_info=$(aws ecr describe-repositories --repository-names ${container_lc})
  if [[ "$repo_container_info" == "" ]]; then
    aws ecr create-repository --repository-name ${container_lc}
  fi
  
  echo "Pushing Image up to AWS"
  docker push ${repo}/${image_and_tag}
  
  # Create compute environment
  EC2_type=ON_DEMAND
  ce_info=$(aws batch describe-compute-environments --compute-environments ${container}_${EC2_type}_CE)
  ce_status=$(echo $ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
  if [[ $ce_status != "VALID" ]]; then
    compute_resources='{"type":"EC2","allocationStrategy":"BEST_FIT_PROGRESSIVE","minvCpus":0,"maxvCpus":10000,"tags":{"Name":"'${container}_Batch'"},"instanceTypes":["c5a","m5a","c5","m5"],"subnets":["'$subnet_useast1a'","'$subnet_useast1b'","'$subnet_useast1c'"],"securityGroupIds":["sg-ff2fd48f","sg-65ff5315"],"ec2KeyPair":"SBTools","instanceRole":"ecsInstanceRole"}'
    aws batch create-compute-environment \
      --compute-environment-name ${container}_${EC2_type}_CE \
      --type MANAGED \
      --state ENABLED \
      --compute-resources $compute_resources \
      --service-role AWSBatchServiceRole
  
    printf "Waiting for Compute Environment to be created"
    while [[ $ce_status != "VALID" ]]; do
  sleep 5
  printf "."
  ce_info=$(aws batch describe-compute-environments --compute-environments ${container}_${EC2_type}_CE)
      ce_status=$(echo $ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
    done
    echo ""
  fi
  
  # Create the job queue
  jq_info=$(aws batch describe-job-queues --job-queues ${container}_${EC2_type}_JQ)
  jq_status=$(echo $jq_info | jq '.jobQueues[0] | .status // ""' | tr -d '"')
  if [[ $jq_status != "VALID" ]]; then
    aws batch create-job-queue \
      --job-queue-name ${container}_${EC2_type}_JQ \
      --state ENABLED \
      --priority 5 \
      --compute-environment-order '{"order":1,"computeEnvironment":"'${container}_${EC2_type}_CE'"}'
  fi
  
  # Register job definition
  job_definition="${container}_JD"
  jd_info=$(aws batch describe-job-definitions --status ACTIVE --job-definition-name $job_definition)
  jd_status=$(echo $jd_info | jq '.jobDefinitions[0] | .status // ""' | tr -d '"')
  if [[ $jd_status != "ACTIVE" ]]; then
    aws batch register-job-definition \
      --job-definition-name $job_definition \
      --type container \
      --container-properties '{"privileged":true,"image":"'$repo/$image_and_tag'","vcpus":'${vCPUs_req}',"memory":'${memory_MiB_req}',"mountPoints":[{"sourceVolume":"SB-EFS","containerPath":"/EFS"}],"volumes":[{"host":{"sourcePath":"/"},"name":"SB-EFS"}],"command":["Ref::container","Ref::run_id","Ref::input_file"]}' \
      --retry-strategy '{"attempts": 10,"evaluateOnExit":[{"onStatusReason" :"Host EC2*","action": "RETRY"},{"onReason" : "*","action": "EXIT"}]}'
  fi
  
  # Make sure the EFS has an access point set up
  efs_id=$(aws efs describe-file-systems | jq -r '.FileSystems[] | select (.Name=="SB-EFS") | .FileSystemId')
  if [[ "$efs_id" == "" ]]; then
    echo "ERROR: SB-EFS does not exist"
    exit
  fi
  ap_id=$(aws efs describe-access-points | jq -r '.AccessPoints[] | select (.Name=="EFS_AP") | .AccessPointId')
  if [[ "$ap_id" == "" ]]; then
    # 1001:1001 is user sb in group sb
    ap_id=$(aws efs create-access-point --file-system-id "$efs_id" --posix-user '{"Uid":1001,"Gid":1001}' --root-directory '{"Path":"/"}' --tags '[{"Key":"Name","Value":"EFS_AP"}]' | jq -r '.AccessPointId')
  fi  
  
  ```
  
  Now when I run it, everything works as it should until getting to the --retry-strategy command line in the "register job definition" section of the script. The following error is subsequently thrown:
  
  ```
  Parameter validation failed:
  Unknown parameter in retryStrategy: "evaluateOnExit", must be one of: attempts
  
  ```
  
  Not sure what the issue is with the "attempts" parameter that is set. Any potential solutions or guidance to solving this error is much appreciated. Thanks!
  
  **Tags:** bash, shell, aws
  
  > **Comments:**
  > **username_1**: Hi @username_0, could you please provide a bit more information about what you're trying to do with this script? Stories help a *lot* with getting good, helpful answers. I can see something suggesting metagenomics, but it's not clear to me what a correct script would do.  
  > **username_2**: My personal approach would be to use the NextFlow AWS pipeline. It would simplify everything, e.g. NextFlow automatically 'Dockerises'  
  > **username_3**: Just a suggestion, but in my experience it's much more readable to use a Python library for such things, try out boto3. I've never tried submitting batch jobs, but I believe it could help in debugging the issue !  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2022-10-31. Score: 0
  
  You didn't say what version of the AWS CLI you're using, but I suspect it's version 1. It looks like the `evaluateOnExit` feature was added in version 2 but was not backported. Installing AWS CLI version 2 resolves the issue:
  
  <https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html>
  
  Note that [Nextflow](https://www.nextflow.io/) also provides built-in support for [AWS Batch](https://www.nextflow.io/docs/latest/aws.html#aws-batch). You'd still need to configure some AWS infrastructure using Nextflow, but it should be a lot less painful than writing/maintaining your own script(s). It's definitely worth checking out if you haven't already.
  
  
  ---
  
  