Answers.Body: <p>Here's an alternative to using a custom task, albeit still rather unappetizing:</p>
  
  <pre><code>workflow groupby {
    input {
      File? nil
    }
  
    Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
    Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]
  
    scatter (y in ["A", "B"]) {
      scatter (p in zip(X,Y)) {
        File? x_or_nil = if (p.right == y) then p.left else nil
      }
      Pair[String,Array[File]] group = (y, select_all(x_or_nil))
    }
  
    output {
      Array[Pair[String,Array[File]]] ans = group
    }
  }
  </code></pre>
  
  <p>(It'll be slightly cleaner with the upcoming <code>None</code> literal instead of dummy <code>nil</code>)</p>
  
  <p>FWIW, <a href="https://github.com/openwdl/wdl/issues/312" rel="nofollow noreferrer">here's a lengthy discussion</a> about the idea of allowing developers to define custom library functions which just run a small task that manipulates data structures (hinting that it needn't necessarily offload to a worker node, with the huge latency that typically entails).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T16:52:00.290Z
Answers.Text: When you use `y in ["A", "B"]` you are doing something that assumes you already know the unique contents of Y. This could be achieved using the `keys()` function but that is also only present in the development version and if the development version is to be used the solution with `collect_by_key()` is more minimalist.
Answers.UserDisplayName: Giulio Genovese
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T20:40:59.630Z
Answers.LastActivityDate: 2020-06-16T20:40:59.630Z
Answers.UserDisplayName: Mike Lin
Answers.Body: <p>It turns out the development version has a function <code>collect_by_key</code> that is particularly helpful for scattering around predefined batches:</p>
  
  <pre><code>Array[String] X  =  ["A",     "A",     "B",     "A",     "B"]
  Array[File] Y = ["file1", "file2", "file3", "file4", "file5"]
  scatter (pair in as_pairs(collect_by_key(zip(X,Y)))) {
    call test {input: X = pair.left, Y = pair.right}
  }
  </code></pre>
  
  <p>This will call task <code>t</code> first with input:</p>
  
  <pre><code>String X = "A"
  Array[File] Y =["file1", "file2", "file4"]
  </code></pre>
  
  <p>And then with input:</p>
  
  <pre><code>String X = "B"
  Array[File] Y =["file3", "file5"]
  </code></pre>
  
  <p>This is a readable solution to this problem allowed by <code>collect_by_key</code>. This function is not very well advertised (I had completely missed it) but kudos to who introduced it in the language. The only drawback is that <code>collect_by_key</code> is not available in version 1.0.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T16:49:03.970Z
Answers.LastActivityDate: 2020-06-17T16:49:03.970Z
Answers.UserDisplayName: Giulio Genovese
Body: <p>Suppose I have two arrays Array[File] X and Array[String] Y both of length N, with Y being a sort of R-like factor that defines batch membership for X, for a total of M batches with M &lt; N. A valid example would be:</p>
  
  <pre><code>Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
  Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]
  </code></pre>
  
  <p>In which case N=5 and M=2. I want to scatter M tasks so that each task takes only the files corresponding to the same batch. In the previous example, one task would take as input:</p>
  
  <pre><code>Array[File] X =["file1", "file2", "file4"]
  String Y = "A"
  </code></pre>
  
  <p>And the other task would take as input:</p>
  
  <pre><code>Array[File] X =["file3", "file5"]
  String Y = "B"
  </code></pre>
  
  <p>Is there an easy way to do this within a WDL workflow in the general case? I understand that there are less elegant ways to do this as its own task but, as this seems like it could be a general purpose scatter/gather problem, an easy solution might be of general interest to the community.</p>
  
Tags: <wdl>
Title: Scatter tasks around predefined batches inside a WDL workflow
ThreadText: # Scatter tasks around predefined batches inside a WDL workflow
  
  (Asked by: username_0 on 2020-06-16)
  
  Suppose I have two arrays Array[File] X and Array[String] Y both of length N, with Y being a sort of R-like factor that defines batch membership for X, for a total of M batches with M < N. A valid example would be:
  
  ```
  Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
  Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]
  
  ```
  
  In which case N=5 and M=2. I want to scatter M tasks so that each task takes only the files corresponding to the same batch. In the previous example, one task would take as input:
  
  ```
  Array[File] X =["file1", "file2", "file4"]
  String Y = "A"
  
  ```
  
  And the other task would take as input:
  
  ```
  Array[File] X =["file3", "file5"]
  String Y = "B"
  
  ```
  
  Is there an easy way to do this within a WDL workflow in the general case? I understand that there are less elegant ways to do this as its own task but, as this seems like it could be a general purpose scatter/gather problem, an easy solution might be of general interest to the community.
  
  **Tags:** wdl
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-16. Score: 1
  
  Here's an alternative to using a custom task, albeit still rather unappetizing:
  
  ```
  workflow groupby {
    input {
      File? nil
    }
  
    Array[File] X = ["file1", "file2", "file3", "file4", "file5"]
    Array[String] Y  =  ["A",     "A",     "B",     "A",     "B"]
  
    scatter (y in ["A", "B"]) {
      scatter (p in zip(X,Y)) {
        File? x_or_nil = if (p.right == y) then p.left else nil
      }
      Pair[String,Array[File]] group = (y, select_all(x_or_nil))
    }
  
    output {
      Array[Pair[String,Array[File]]] ans = group
    }
  }
  
  ```
  
  (It'll be slightly cleaner with the upcoming `None` literal instead of dummy `nil`)
  
  FWIW, [here's a lengthy discussion](https://github.com/openwdl/wdl/issues/312) about the idea of allowing developers to define custom library functions which just run a small task that manipulates data structures (hinting that it needn't necessarily offload to a worker node, with the huge latency that typically entails).
  
  > **Comments:**
  > **username_0**: When you use `y in ["A", "B"]` you are doing something that assumes you already know the unique contents of Y. This could be achieved using the `keys()` function but that is also only present in the development version and if the development version is to be used the solution with `collect_by_key()` is more minimalist.  
  
  ---
  
  ### Answer by username_0 on 2020-06-17. Score: 0
  
  It turns out the development version has a function `collect_by_key` that is particularly helpful for scattering around predefined batches:
  
  ```
  Array[String] X  =  ["A",     "A",     "B",     "A",     "B"]
  Array[File] Y = ["file1", "file2", "file3", "file4", "file5"]
  scatter (pair in as_pairs(collect_by_key(zip(X,Y)))) {
    call test {input: X = pair.left, Y = pair.right}
  }
  
  ```
  
  This will call task `t` first with input:
  
  ```
  String X = "A"
  Array[File] Y =["file1", "file2", "file4"]
  
  ```
  
  And then with input:
  
  ```
  String X = "B"
  Array[File] Y =["file3", "file5"]
  
  ```
  
  This is a readable solution to this problem allowed by `collect_by_key`. This function is not very well advertised (I had completely missed it) but kudos to who introduced it in the language. The only drawback is that `collect_by_key` is not available in version 1.0.
  
  
  ---
  
  