Answers: <p>Yes, you can start ECS Fargate tasks from Step Functions in the way you've described.</p>
  <p>Step Functions allows you to pass <code>Parameters</code> to the underlying service API.</p>
  <p><a href="https://docs.aws.amazon.com/step-functions/latest/dg/connect-parameters.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/step-functions/latest/dg/connect-parameters.html</a></p>
  <p><em>[sidebar: that doc also specifies how to pass state inputs, which you've stated is a requirement]</em></p>
  <p>For ECS Fargate, a single state machine with no unique configuration could look something like this:</p>
  <pre><code> &quot;States&quot;: {
      &quot;${name}&quot;: {
        &quot;Type&quot;: &quot;Task&quot;,
        &quot;Resource&quot;: &quot;arn:aws:states:::ecs:runTask.sync&quot;,
        &quot;Parameters&quot;: {
          &quot;LaunchType&quot;: &quot;FARGATE&quot;,
          &quot;Cluster&quot;: &quot;${cluster}&quot;,
          &quot;TaskDefinition&quot;: &quot;${task_definition}&quot;,
          &quot;NetworkConfiguration&quot;: {
            &quot;AwsvpcConfiguration&quot;: {
              &quot;Subnets&quot;: [&quot;${subnet_1}&quot;, &quot;${subnet_2}&quot;],
              &quot;SecurityGroups&quot;: [&quot;${security_group}&quot;],
              &quot;AssignPublicIp&quot;: &quot;DISABLED&quot;
            }
          }
        },
  </code></pre>
  <p>Now, you've asked about passing each task slightly different parameter values...</p>
  <p>Thankfully, the ECS <code>RunTask</code> API has a helpful <code>overrides: { containerOverrides: [] }</code> parameter.</p>
  <p><a href="https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html</a></p>
  <p>Overriding the <code>command</code> (for task execution) or <code>environment</code> (for OS variables) should provide the customization you require without needing multiple ECS Task Definitions.</p>
  <p>So, add each distinct task to your state machine, each ingesting state inputs and with its own unique override, and that should do it!</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-06-23T17:10:47.867Z
Answers: 2020-06-23T17:10:47.867Z
Answers: Woodland
Body: <p>I am building a service that has two independent steps. The first step is a data transformation step that has to happen and is common to all downstream processes, then the second is starting a few dozen independent parallel processes that use the same input data, save for a few different runtime specified parameters.</p>
  <p>I know I can use a step function to set off the data transformation step and wait for it to complete before starting off a single fargate task, but can I get it to start a few dozen (or more) in parallel using a common docker object but pass each of those runs slightly different parameter values.</p>
  <p>I want to use fargate because we are a small team and I don't want to have to spend team resources on maintaining compute resources, and I can't use AWS Lambda because we expect the run times will often exceed 15 minutes, and there isn't an easy or obvious way to break the problem down any further to be certain the jobs will be able to execute within 15 minutes.</p>
  
Tags: <docker><amazon-web-services><fargate><step-functions>
Title: Can I use AWS Step Functions to launch many different fargate 'runs'?
UserDisplayName: Brad Davis
ThreadText: # Can I use AWS Step Functions to launch many different fargate 'runs'?
  
  (Asked by: username_0 on 2020-06-23)
  
  I am building a service that has two independent steps. The first step is a data transformation step that has to happen and is common to all downstream processes, then the second is starting a few dozen independent parallel processes that use the same input data, save for a few different runtime specified parameters.
  
  I know I can use a step function to set off the data transformation step and wait for it to complete before starting off a single fargate task, but can I get it to start a few dozen (or more) in parallel using a common docker object but pass each of those runs slightly different parameter values.
  
  I want to use fargate because we are a small team and I don't want to have to spend team resources on maintaining compute resources, and I can't use AWS Lambda because we expect the run times will often exceed 15 minutes, and there isn't an easy or obvious way to break the problem down any further to be certain the jobs will be able to execute within 15 minutes.
  
  **Tags:** docker, amazon-web-services, fargate, step-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-23. Score: 2 | Accepted answer
  
  Yes, you can start ECS Fargate tasks from Step Functions in the way you've described.
  
  Step Functions allows you to pass `Parameters` to the underlying service API.
  
  <https://docs.aws.amazon.com/step-functions/latest/dg/connect-parameters.html>
  
  *[sidebar: that doc also specifies how to pass state inputs, which you've stated is a requirement]*
  
  For ECS Fargate, a single state machine with no unique configuration could look something like this:
  
  ```
   "States": {
      "${name}": {
        "Type": "Task",
        "Resource": "arn:aws:states:::ecs:runTask.sync",
        "Parameters": {
          "LaunchType": "FARGATE",
          "Cluster": "${cluster}",
          "TaskDefinition": "${task_definition}",
          "NetworkConfiguration": {
            "AwsvpcConfiguration": {
              "Subnets": ["${subnet_1}", "${subnet_2}"],
              "SecurityGroups": ["${security_group}"],
              "AssignPublicIp": "DISABLED"
            }
          }
        },
  
  ```
  
  Now, you've asked about passing each task slightly different parameter values...
  
  Thankfully, the ECS `RunTask` API has a helpful `overrides: { containerOverrides: [] }` parameter.
  
  <https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html>
  
  Overriding the `command` (for task execution) or `environment` (for OS variables) should provide the customization you require without needing multiple ECS Task Definitions.
  
  So, add each distinct task to your state machine, each ingesting state inputs and with its own unique override, and that should do it!
  
  
  ---
  
  