Answers.Body: <p>Optional outputs should work in WDL 1.0. Here is an example that runs without issue through cromwell (version 74).</p>
  <p>Note that for the second call in the scatter the input file needs to be converted to from an optional to an non optional value. Since this only gets run if this value is defined, this conversion can safely be done using the <code>select_first</code> function (which returns the first defined value in a given list of optionals).</p>
  <pre><code>version 1.0
  
  workflow test {
      input {
          File filelist = &quot;files.txt&quot;
      }
      Array[String] files = read_lines(filelist)
  
      scatter (file in  files) {
           call blah as b1 {input: in=file}
           if (defined(b1.out)) {
               call blah as b2 {input: in=select_first([b1.out])}
           }
      }
  }
  
  
  task blah {
      input {
          File in
      }
      String outPath = basename(in,'.txt') + '_blah.txt'
  
      command {
          if $(grep -q &quot;yay&quot; ~{in})
            then
               echo yay &gt; ~{outPath}
          fi
      }
  
      output {
          File? out = outPath
      }
  }
  </code></pre>
  <p>files.txt looks like this:</p>
  <pre><code>test1.txt
  test2.txt
  </code></pre>
  <p>With test1.txt containing <code>yay</code> and test2.txt containing <code>nay</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-05T09:45:47.247Z
Answers.LastActivityDate: 2022-05-05T09:45:47.247Z
Answers.UserDisplayName: DavyCats
Body: <p>I'm trying to adapt an existing workflow to make it more robust to failures. Below is a simplified example of the workflow.</p>
  <pre><code>workflow main {
     File filelist=&quot;list_of_input_files.txt&quot;
     Array[String] files=read_lines(filelist)
     scatter(file in files) {
        call task1 { input: in=file }
        if(defined(task1.out)) { call task2 { input: in=task1.out } }
        if(defined(task2.out)) { call task2 { input: in=task2.out }
     }
  }
  
  task task1 {
     File in
     String out1 = basename(in,'.txt') + '_1.txt'
     command { if(...) { ./script1.sh <span class="math-container">${in} &gt; $</span>{out1} } }
     output { File? out = out1 }
     runtime { ... }
  }
  
  task task2 {
     File in
     String out2 = basename(in,'.txt') + '_2.txt'
     command { if(...) {./script2.sh <span class="math-container">${in} &gt; $</span>{out2} } }
     output { File? out = out2 }
     runtime { ... }
  }
  
  task task3 {
     File in
     String out3 = basename(in,'.txt') + '_3.txt'
     command { if(...) {./script3.sh <span class="math-container">${in} &gt; $</span>{out3} } }
     output { File? out = out3 }
     runtime {...}
  }
  </code></pre>
  <p>with my list_of_input_files.txt looking like</p>
  <pre><code>firstfile.txt
  secondfile.txt
  thirdfile.txt
  ...
  </code></pre>
  <p>Obviously this is oversimplified, but I am trying to catch failed blocks by making the output of each task optional (using <code>File?</code>) and then having an <code>if(defined(taskX.out))</code> condition to call the next task. My reasoning is that if something fails (represented by the <code>if(...)</code> within the command block of each task), then the <code>out</code> file will never be created in that task and so won't exist as input for the next task and thus <code>if(defined(taskX.out))</code> will resolve as false and skip the next task (and therefore all the subsequent tasks).</p>
  <p>With current testing, the first file in the list satisfies the <code>if(...)</code> conditions of all the tasks and succeeds in giving me the final output file <code>firstfile_1_2_3.txt</code>. However, I am getting a <code>Required file output 'secondfile_1.txt' does not exist'</code> error message because it fails the <code>if(...)</code> condition in <code>task1</code> and so the output file is never created.</p>
  <p>From what I've read in the <a href="https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#declarations" rel="nofollow noreferrer">WDL v1.1 spec</a>, optional <code>File</code> outputs are allowed and <a href="https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#file-directory-and-optional-outputs" rel="nofollow noreferrer">are undefined if the files they point to don't exist</a>. However, after much confused googling, I've realised it's probably because I'm using <a href="https://github.com/broadinstitute/cromwell" rel="nofollow noreferrer">Cromwell</a>, which is stuck with WDL v1.0.</p>
  <p>My question is whether this should be working in WDL v1.0 and, if not, is my logic correct and should this (psuedo-)code work with WDL v1.1? Is there a way to achieve what I'm trying to do using pure WDL v1.0?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-04T06:52:01.603Z
Comments.Text: @M__ Thanks - my confusion is based around the different specs. It seems that the features I'm trying to use (optional files) don't seem to be supported in the spec that Cromwell uses.
Comments.UserDisplayName: tiit_helimut
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-04T06:55:54.347Z
Comments.Text: @DavyCats Thanks for the comment! I think I've made the mistake to assume that Cromwell incorporates the latest drafts, but it actually uses WDL 1.0 and so these features are not supported. Despite the WDL 1.1 spec being > 1 year old, I can't see any Cromwell implementation based around it yet.
Comments.UserDisplayName: tiit_helimut
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-09T18:59:04.813Z
Comments.Text: I stayed clear of WDL because seemed to over complicate things. Can re-write this as a bash script but would need to study the pipeline a bit more.
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-09T19:00:10.357Z
Comments.Text: I think the thing to do is contact Broad Institute directly because this appears to be a version issue pertaining the upgrade from 1.0 to 1.1
Comments.UserDisplayName: M__
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-11T11:33:05.107Z
Comments.Text: Your WDL code looks more like draft-2 WDL than 1.0 or 1.1 WDL. In WDL 1.0/1.1 there is a separate input section in tasks and workflows, and the version must be specified on the first line. In any case, optional file outputs should be supported in all versions.
Comments.UserDisplayName: DavyCats
Tags: <wdl><workflow-management>
Title: Handling failed blocks within scattered WDL workflow
ThreadText: # Handling failed blocks within scattered WDL workflow
  
  (Asked by: username_0 on 2022-03-08)
  
  I'm trying to adapt an existing workflow to make it more robust to failures. Below is a simplified example of the workflow.
  
  ```
  workflow main {
     File filelist="list_of_input_files.txt"
     Array[String] files=read_lines(filelist)
     scatter(file in files) {
        call task1 { input: in=file }
        if(defined(task1.out)) { call task2 { input: in=task1.out } }
        if(defined(task2.out)) { call task2 { input: in=task2.out }
     }
  }
  
  task task1 {
     File in
     String out1 = basename(in,'.txt') + '_1.txt'
     command { if(...) { ./script1.sh ${in} > ${out1} } }
     output { File? out = out1 }
     runtime { ... }
  }
  
  task task2 {
     File in
     String out2 = basename(in,'.txt') + '_2.txt'
     command { if(...) {./script2.sh ${in} > ${out2} } }
     output { File? out = out2 }
     runtime { ... }
  }
  
  task task3 {
     File in
     String out3 = basename(in,'.txt') + '_3.txt'
     command { if(...) {./script3.sh ${in} > ${out3} } }
     output { File? out = out3 }
     runtime {...}
  }
  
  ```
  
  with my list\_of\_input\_files.txt looking like
  
  ```
  firstfile.txt
  secondfile.txt
  thirdfile.txt
  ...
  
  ```
  
  Obviously this is oversimplified, but I am trying to catch failed blocks by making the output of each task optional (using `File?`) and then having an `if(defined(taskX.out))` condition to call the next task. My reasoning is that if something fails (represented by the `if(...)` within the command block of each task), then the `out` file will never be created in that task and so won't exist as input for the next task and thus `if(defined(taskX.out))` will resolve as false and skip the next task (and therefore all the subsequent tasks).
  
  With current testing, the first file in the list satisfies the `if(...)` conditions of all the tasks and succeeds in giving me the final output file `firstfile_1_2_3.txt`. However, I am getting a `Required file output 'secondfile_1.txt' does not exist'` error message because it fails the `if(...)` condition in `task1` and so the output file is never created.
  
  From what I've read in the [WDL v1.1 spec](https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#declarations), optional `File` outputs are allowed and [are undefined if the files they point to don't exist](https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#file-directory-and-optional-outputs). However, after much confused googling, I've realised it's probably because I'm using [Cromwell](https://github.com/broadinstitute/cromwell), which is stuck with WDL v1.0.
  
  My question is whether this should be working in WDL v1.0 and, if not, is my logic correct and should this (psuedo-)code work with WDL v1.1? Is there a way to achieve what I'm trying to do using pure WDL v1.0?
  
  **Tags:** wdl, workflow-management
  
  > **Comments:**
  > **username_1**: I stayed clear of WDL because seemed to over complicate things. Can re-write this as a bash script but would need to study the pipeline a bit more.  
  > **username_1**: I think the thing to do is contact Broad Institute directly because this appears to be a version issue pertaining the upgrade from 1.0 to 1.1  
  > **username_2**: Your WDL code looks more like draft-2 WDL than 1.0 or 1.1 WDL. In WDL 1.0/1.1 there is a separate input section in tasks and workflows, and the version must be specified on the first line. In any case, optional file outputs should be supported in all versions.  
  > **username_0**: @username_1 Thanks - my confusion is based around the different specs. It seems that the features I'm trying to use (optional files) don't seem to be supported in the spec that Cromwell uses.  
  > **username_0**: @username_2 Thanks for the comment! I think I've made the mistake to assume that Cromwell incorporates the latest drafts, but it actually uses WDL 1.0 and so these features are not supported. Despite the WDL 1.1 spec being > 1 year old, I can't see any Cromwell implementation based around it yet.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-05. Score: 1
  
  Optional outputs should work in WDL 1.0. Here is an example that runs without issue through cromwell (version 74).
  
  Note that for the second call in the scatter the input file needs to be converted to from an optional to an non optional value. Since this only gets run if this value is defined, this conversion can safely be done using the `select_first` function (which returns the first defined value in a given list of optionals).
  
  ```
  version 1.0
  
  workflow test {
      input {
          File filelist = "files.txt"
      }
      Array[String] files = read_lines(filelist)
  
      scatter (file in  files) {
           call blah as b1 {input: in=file}
           if (defined(b1.out)) {
               call blah as b2 {input: in=select_first([b1.out])}
           }
      }
  }
  
  task blah {
      input {
          File in
      }
      String outPath = basename(in,'.txt') + '_blah.txt'
  
      command {
          if $(grep -q "yay" ~{in})
            then
               echo yay > ~{outPath}
          fi
      }
  
      output {
          File? out = outPath
      }
  }
  
  ```
  
  files.txt looks like this:
  
  ```
  test1.txt
  test2.txt
  
  ```
  
  With test1.txt containing `yay` and test2.txt containing `nay`.
  
  
  ---
  
  