Answers: <p>I hope this helps. We had a similar issue and solved it using the below method.</p>
  
  <p>Our project contains about 12 or so different components (programs) that we wanted to build in parallel. Because some take longer than others, we decided to break them up into 4 individual "tracks" that would run in parallel. To begin, assume we have an array where each element is a new component object. The object has some properties, like name for the name of the component and track for which track, 1, 2, 3, or 4, that the component is assigned to. I'm not focusing on this part as it's probably not relevant to what you're trying to do. </p>
  
  <p>First we define our <code>createParallelStage</code> function which iterates over an ArrayList of components, setting them up to build.</p>
  
  <pre><code>def createParallelStage(ArrayList components) {
      return{
          // iterate over list of components objects
          components.each { component -&gt;
              dir(buildFolder) {
                  // build the component
                  sh "./build.sh ${component}"
  
              }
          }
      }
  }
  </code></pre>
  
  <p>We'll use a list of component objects called BuildComponents, iterate over them, and add them to <code>parallelGroups[i]</code> where <code>i</code> is the value of the objects <code>track</code> property</p>
  
  <pre><code>BuildComponents.each { component -&gt;
      parallelGroups[component.track].add(component.name)
      echo "${component.name} is going to parallel track ${component.track}"
      // example: IPCService is going to parallel track 3
      }
  }
  </code></pre>
  
  <p>We're now left with iterating over <code>parallelGroups</code>, passing each element to <code>createParallelStage()</code> while adding it to the <code>parallelStages</code> ArrayList.</p>
  
  <pre><code>parallelGroups.each{i, component -&gt;
      parallelStages.put(i, createParallelStage(component))
  }
  </code></pre>
  
  <p>We then set <code>failFast</code> to true, so if any of our components fail, the whole build job will fail, and then pass our list of <code>parallelStages</code> to the <code>parallel()</code> function.</p>
  
  <pre><code>parallelStages.failFast = true
  parallel(parallelStages)
  </code></pre>
  
  <p>I hope this was specific to the question you were asking. I wish I could be a little more detailed, but then it might be too specific of a solution to work for you.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-12-31T15:26:27.783Z
Answers: 2019-12-31T15:26:27.783Z
Answers: Argyle
Answers: <p>This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:</p>
  
  <ol>
  <li>We have a shared library where we keep our scripts ins <code>./vars/someScriptName.groovy</code></li>
  <li>We created a <code>groovy</code> script that generates the stages.</li>
  </ol>
  
  <p>Here's a working pipeline code for you to try it out:</p>
  
  <pre><code>def generateITParallelStages(body)
  {
      def config = [:]
      config.stages = []
  
      body.resolveStrategy = Closure.DELEGATE_FIRST
      body.delegate = config
      body()
  
      def buildStages = [:]
      config.stages.each { String stage, closure -&gt;
          // default settings, if you want to have any
          def settings = [:]
          settings.type = 'online'; // if you don't define the type, it will be 'online' by default
          settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default
  
          closure.resolveStrategy = Closure.DELEGATE_FIRST
          closure.delegate = settings
          closure()
  
          buildStages.put(stage, prepareStage(stage, settings))
      }
  
      return buildStages
  }
  
  def prepareStage(String stage, Map settings)
  {
      return {
          print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
      }
  }
  
  pipeline {
      agent any
      stages {
          stage('Add regression tests') {
              steps {
                  script {
                      parallel generateITParallelStages {
                          stages = [
                                  webgoat: {
                                      type = 'offline'
                                      suite = 'open-source'
                                  },
                                  'juice-shop': {
                                      type = 'offline'
                                      suite = 'open-source'
                                  },
                                  'our-website': {
                                      type = 'online'
                                      suite = 'internal'
                                  }
                          ]
                      }
                  }
              }
          }
      }
  }
  </code></pre>
  
  <p>And here's the result:</p>
  
  <p><a href="https://i.sstatic.net/am8XK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/am8XK.png" alt="pipeline-view"></a></p>
  
  <p>This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this). </p>
  
  <p>Other people seem to be having a hard time figuring this out as well - <a href="https://issues.jenkins-ci.org/browse/JENKINS-53032" rel="nofollow noreferrer">JENKINS-53032</a>. If you feel the same make some noise :)</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-01-12T23:15:34.153Z
Answers: I would like to set a docker agent on the stage, but I am getting an error that seems to indicate I can only use step interface classes as operations in the stage execution. Where would I define this setting? I have tried to find a JavaDoc that shows the underlying classes of the API, but have come up empty.
Answers: Jörn Guy Süß
Answers: CC BY-SA 4.0
Answers: 2020-01-12T23:19:29.950Z
Answers: You would need to set the `Add regression tests` to run on `none` agent and then have in the `prepareStage` a `node(settings.agent) { ... }` which starts the parallel stage.
Answers: tftd
Answers: CC BY-SA 4.0
Answers: 2020-01-13T03:24:41.763Z
Answers: Sorry for being such a nuisance. Is there a way I can add a sequence to the stage? I have tried calling the `stages {}` closure, but it is not valid. I assume I am in a scripted environment now, guessing from your previous instruction, which works great!
Answers: Jörn Guy Süß
Answers: CC BY-SA 4.0
Answers: 2020-01-13T13:44:26.940Z
Answers: That's correct - while in `prepareStage` you are writing `scripted` pipeline code. This means, there is no `post {}`, no `stages {}` and other "sugar" from the declarative pipes. I assume you want to add a few stages on top of the running parallel stage (i.e. have `unit-test`, `integration-test` in `juice-shop`). To do that, just have `node(settings.agent) { stage('ut') {...} stage('it') {...} }`. Checkout https://jenkins.io/doc/book/pipeline/#scripted-pipeline-fundamentals. If that's not what you want - please open up a new question and I'll try to answer.
Answers: tftd
Answers: CC BY-SA 4.0
Answers: 2019-12-27T16:09:42.330Z
Answers: 2019-12-27T18:00:21.757Z
Answers: 2019-12-27T18:00:21.757Z
Answers: tftd
Body: <p>In declarative pipelines, Jenkins allows the definition of <a href="https://jenkins.io/doc/book/pipeline/syntax/#parallel" rel="noreferrer">parallel stages</a>. It further allows <a href="https://jenkins.io/doc/book/pipeline/syntax/#script" rel="noreferrer">scripted pipeline general purpose scripts</a> to create and manipulate the artifacts of the declarative pipeline. For example, it can create <code>stages</code> dynamically.</p>
  
  <p>I would like to know how I can <em>dynamically</em> create <em>parallel executing stages</em>. Please note I am not talking about <em>steps</em> or actions, which are solved here:</p>
  
  <p><a href="https://devops.stackexchange.com/questions/3073/how-to-properly-achieve-dynamic-parallel-action-with-a-declarative-pipeline">How to properly achieve dynamic parallel action with a declarative pipeline?</a></p>
  
  <p>Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the <code>parallel</code> construct at the <code>stage</code> level. As a result, three stages that run parallel should be generated by the program.</p>
  
  <pre class="lang-java prettyprint-override"><code>pipeline {
      agent any
      stages {
          stage('Add regression tests') {
              steps {
                  addStage('offline','open-source','webgoat')
                  addStage('offline','open-source','juice-shop')
                  addStage('online','internal','our-website')
              }
          }
      }
  }
  
  final void addStage(final String type, final String suite,final String application) {
      script {
          stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {
  
              runTestOnApplication(type,suite,application)
  
          }
      }
  }
  
  final void runTestOnApplication(final String type, final String suite,final String application) {
      labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
  }
  
  </code></pre>
  
Comments: CC BY-SA 4.0
Comments: 2019-12-27T16:25:55.490Z
Comments: @tftd Do you know whether there is a related Jira Jenkins issue for this? If not, perhaps you could create it and link it to this Q&A. Perhaps creating an issue on the Jenkins Jira board could create awareness to make it less complicated?
Comments: 030
Comments: CC BY-SA 4.0
Comments: 2019-12-27T16:34:17.663Z
Comments: @030 It looks like [JENKINS-53032](https://issues.jenkins-ci.org/browse/JENKINS-53032) asking for something similar.
Comments: tftd
Comments: CC BY-SA 4.0
Comments: 2019-12-27T16:37:48.477Z
Comments: @tftd Well done. Perhaps you could add a link to this ticket to your answer and a link to your answer to the jenkins ticket?
Comments: 030
Tags: <jenkins-pipeline><jenkinsfile>
Title: How to define dynamic parallel stages in a Jenkinsfile?
UserDisplayName: Jörn Guy Süß
ThreadText: # How to define dynamic parallel stages in a Jenkinsfile?
  
  (Asked by: username_0 on 2019-11-22)
  
  In declarative pipelines, Jenkins allows the definition of [parallel stages](https://jenkins.io/doc/book/pipeline/syntax/#parallel). It further allows [scripted pipeline general purpose scripts](https://jenkins.io/doc/book/pipeline/syntax/#script) to create and manipulate the artifacts of the declarative pipeline. For example, it can create `stages` dynamically.
  
  I would like to know how I can *dynamically* create *parallel executing stages*. Please note I am not talking about *steps* or actions, which are solved here:
  
  [How to properly achieve dynamic parallel action with a declarative pipeline?](https://devops.stackexchange.com/questions/3073/how-to-properly-achieve-dynamic-parallel-action-with-a-declarative-pipeline)
  
  Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the `parallel` construct at the `stage` level. As a result, three stages that run parallel should be generated by the program.
  
  ```java
  pipeline {
      agent any
      stages {
          stage('Add regression tests') {
              steps {
                  addStage('offline','open-source','webgoat')
                  addStage('offline','open-source','juice-shop')
                  addStage('online','internal','our-website')
              }
          }
      }
  }
  
  final void addStage(final String type, final String suite,final String application) {
      script {
          stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {
  
              runTestOnApplication(type,suite,application)
  
          }
      }
  }
  
  final void runTestOnApplication(final String type, final String suite,final String application) {
      labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
  }
  
  ```
  
  **Tags:** jenkins-pipeline, jenkinsfile
  
  > **Comments:**
  > **username_1**: @username_2 Do you know whether there is a related Jira Jenkins issue for this? If not, perhaps you could create it and link it to this Q&A. Perhaps creating an issue on the Jenkins Jira board could create awareness to make it less complicated?  
  > **username_2**: @username_1 It looks like [JENKINS-53032](https://issues.jenkins-ci.org/browse/JENKINS-53032) asking for something similar.  
  > **username_1**: @username_2 Well done. Perhaps you could add a link to this ticket to your answer and a link to your answer to the jenkins ticket?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-12-27. Score: 3 | Accepted answer
  
  This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:
  
  1. We have a shared library where we keep our scripts ins `./vars/someScriptName.groovy`
  2. We created a `groovy` script that generates the stages.
  
  Here's a working pipeline code for you to try it out:
  
  ```
  def generateITParallelStages(body)
  {
      def config = [:]
      config.stages = []
  
      body.resolveStrategy = Closure.DELEGATE_FIRST
      body.delegate = config
      body()
  
      def buildStages = [:]
      config.stages.each { String stage, closure ->
          // default settings, if you want to have any
          def settings = [:]
          settings.type = 'online'; // if you don't define the type, it will be 'online' by default
          settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default
  
          closure.resolveStrategy = Closure.DELEGATE_FIRST
          closure.delegate = settings
          closure()
  
          buildStages.put(stage, prepareStage(stage, settings))
      }
  
      return buildStages
  }
  
  def prepareStage(String stage, Map settings)
  {
      return {
          print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
      }
  }
  
  pipeline {
      agent any
      stages {
          stage('Add regression tests') {
              steps {
                  script {
                      parallel generateITParallelStages {
                          stages = [
                                  webgoat: {
                                      type = 'offline'
                                      suite = 'open-source'
                                  },
                                  'juice-shop': {
                                      type = 'offline'
                                      suite = 'open-source'
                                  },
                                  'our-website': {
                                      type = 'online'
                                      suite = 'internal'
                                  }
                          ]
                      }
                  }
              }
          }
      }
  }
  
  ```
  
  And here's the result:
  
  [![pipeline-view](https://i.sstatic.net/am8XK.png)](https://i.sstatic.net/am8XK.png)
  
  This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this).
  
  Other people seem to be having a hard time figuring this out as well - [JENKINS-53032](https://issues.jenkins-ci.org/browse/JENKINS-53032). If you feel the same make some noise :)
  
  > **Comments:**
  > **username_0**: I would like to set a docker agent on the stage, but I am getting an error that seems to indicate I can only use step interface classes as operations in the stage execution. Where would I define this setting? I have tried to find a JavaDoc that shows the underlying classes of the API, but have come up empty.  
  > **username_2**: You would need to set the `Add regression tests` to run on `none` agent and then have in the `prepareStage` a `node(settings.agent) { ... }` which starts the parallel stage.  
  > **username_0**: Sorry for being such a nuisance. Is there a way I can add a sequence to the stage? I have tried calling the `stages {}` closure, but it is not valid. I assume I am in a scripted environment now, guessing from your previous instruction, which works great!  
  > **username_2**: That's correct - while in `prepareStage` you are writing `scripted` pipeline code. This means, there is no `post {}`, no `stages {}` and other "sugar" from the declarative pipes. I assume you want to add a few stages on top of the running parallel stage (i.e. have `unit-test`, `integration-test` in `juice-shop`). To do that, just have `node(settings.agent) { stage('ut') {...} stage('it') {...} }`. Checkout https://jenkins.io/doc/book/pipeline/#scripted-pipeline-fundamentals. If that's not what you want - please open up a new question and I'll try to answer.  
  
  ---
  
  ### Answer by username_3 on 2019-12-31. Score: 1
  
  I hope this helps. We had a similar issue and solved it using the below method.
  
  Our project contains about 12 or so different components (programs) that we wanted to build in parallel. Because some take longer than others, we decided to break them up into 4 individual "tracks" that would run in parallel. To begin, assume we have an array where each element is a new component object. The object has some properties, like name for the name of the component and track for which track, 1, 2, 3, or 4, that the component is assigned to. I'm not focusing on this part as it's probably not relevant to what you're trying to do.
  
  First we define our `createParallelStage` function which iterates over an ArrayList of components, setting them up to build.
  
  ```
  def createParallelStage(ArrayList components) {
      return{
          // iterate over list of components objects
          components.each { component ->
              dir(buildFolder) {
                  // build the component
                  sh "./build.sh ${component}"
  
              }
          }
      }
  }
  
  ```
  
  We'll use a list of component objects called BuildComponents, iterate over them, and add them to `parallelGroups[i]` where `i` is the value of the objects `track` property
  
  ```
  BuildComponents.each { component ->
      parallelGroups[component.track].add(component.name)
      echo "${component.name} is going to parallel track ${component.track}"
      // example: IPCService is going to parallel track 3
      }
  }
  
  ```
  
  We're now left with iterating over `parallelGroups`, passing each element to `createParallelStage()` while adding it to the `parallelStages` ArrayList.
  
  ```
  parallelGroups.each{i, component ->
      parallelStages.put(i, createParallelStage(component))
  }
  
  ```
  
  We then set `failFast` to true, so if any of our components fail, the whole build job will fail, and then pass our list of `parallelStages` to the `parallel()` function.
  
  ```
  parallelStages.failFast = true
  parallel(parallelStages)
  
  ```
  
  I hope this was specific to the question you were asking. I wish I could be a little more detailed, but then it might be too specific of a solution to work for you.
  
  
  ---
  
  