Answers.Body: <p>You could use the <a href="https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#string-substring-string-string" rel="nofollow noreferrer"><code>sub</code></a> function to obtain the chromosome from the globbed file names and then use that to retrieve the correct LD matrices from a Map (see <a href="https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#mapstring-string-read_mapstringfile" rel="nofollow noreferrer"><code>read_map</code></a> or <a href="https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#mixed-read_jsonstringfile" rel="nofollow noreferrer"><code>read_json</code></a>).</p>
  
  <p>eg.:</p>
  
  <pre><code>Map[String, String] ld_matrices = read_json(file_of_ld_matrices_by_chr)
  
  scatter(gwas_for_chr in split.gwas_by_chr) {
      String chr = sub(sub(gwas_for_chr, prefix, ""), "\.assoc$", "")
  
      call sbayes.run {
          input:
              GCTB = gctb_executable_path,
              gwas = gwas_for_chr,
              ld_matrix = ld_matrices[chr],
              output_prefix = out
      }
  }
  </code></pre>
  
  <p>The <code>file_of_ld_matrices_by_chr</code> would have to look like this:</p>
  
  <pre><code>{
    "chr1": "/path/to/file1",
    "chr2": "/path/to/file2"
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T15:20:41.120Z
Answers.Text: This works perfectly.
Answers.UserDisplayName: Vivek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T14:59:49.550Z
Answers.LastActivityDate: 2020-04-27T14:59:49.550Z
Answers.UserDisplayName: DavyCats
Body: <p>I'm trying to implement a polygenic score pipeline in WDL and I'm quite new to this pipeline management.</p>
  
  <p>The first step of the pipeline takes a file of GWAS summary statistics and splits it by chromosome.</p>
  
  <p>The corresponding wdl task:</p>
  
  <pre><code>task split {
      input {
          File gwas
          String output_prefix
      }
  
      command {
          ./splitGwas -i <span class="math-container">${gwas} -o $</span>{output_prefix}
      }
  
      output {
          Array [File] gwas_by_chr = glob("${output_prefix}_*.assoc")
      }
  }
  </code></pre>
  
  <p>The next part of the process is to compute posterior effects of SNPs by chromosome. The inputs to this task are going to be the files split by chromosome in the previous step and an LD matrix for the chromosome.</p>
  
  <p>The corresponding wdl call would be something like this:</p>
  
  <pre><code>Array [File] ld_matrices = read_lines(file_of_ld_matrices_by_chr)
  
  scatter(pair in zip(split.gwas_by_chr, ld_matrices)) {
      call sbayes.run {
          input:
              GCTB = gctb_executable_path,
              gwas = pair.left,
              ld_matrix = pair.right,
              output_prefix = out
      }
  }
  </code></pre>
  
  <p>Since I'm grabbing the split files using glob in the previous step, I do not particularly know which element in the array corresponds to which chromosome. So I would be making a mistake if I zip the summary stats for one chromosome with the LD matrix corresponding to a different chromosome.</p>
  
  <p>The programmatic way would be to iterate the scatter over chromosome name but then I would lose the ability to specify the dependency between the two tasks.</p>
  
  <p>Is there a better way to do this?</p>
  
  <p>Cross posted on biostars (<a href="https://www.biostars.org/p/434582/" rel="nofollow noreferrer">https://www.biostars.org/p/434582/</a>) so pardon me if that is discouraged.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-27T11:47:59.223Z
Comments.Text: Two questions: 1. Would it be possible for `file_of_ld_matrices_by_chr` to be in JSON format, instead of just a list a list of file paths? 2. What does the name of the file produced by `splitGwas` look like (ie. what's in between the prefix and the `.assoc`?
Comments.UserDisplayName: DavyCats
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-27T14:31:20.837Z
Comments.Text: 1. Yes, I can put the LD matrices in JSON
  2. The file names produced by split GWAS have the names ${prefix}_chromosome.assoc so if the user supplied prefix string is T2D as in type 2 diabetes, the outputs are T2D_1.assoc, T2D_2.assoc ... T2D_22.assoc
Comments.UserDisplayName: Vivek
Tags: <wdl>
Title: Matching files by chromosome during wdl scatter gather
ThreadText: # Matching files by chromosome during wdl scatter gather
  
  (Asked by: username_0 on 2020-04-27)
  
  I'm trying to implement a polygenic score pipeline in WDL and I'm quite new to this pipeline management.
  
  The first step of the pipeline takes a file of GWAS summary statistics and splits it by chromosome.
  
  The corresponding wdl task:
  
  ```
  task split {
      input {
          File gwas
          String output_prefix
      }
  
      command {
          ./splitGwas -i ${gwas} -o ${output_prefix}
      }
  
      output {
          Array [File] gwas_by_chr = glob("${output_prefix}_*.assoc")
      }
  }
  
  ```
  
  The next part of the process is to compute posterior effects of SNPs by chromosome. The inputs to this task are going to be the files split by chromosome in the previous step and an LD matrix for the chromosome.
  
  The corresponding wdl call would be something like this:
  
  ```
  Array [File] ld_matrices = read_lines(file_of_ld_matrices_by_chr)
  
  scatter(pair in zip(split.gwas_by_chr, ld_matrices)) {
      call sbayes.run {
          input:
              GCTB = gctb_executable_path,
              gwas = pair.left,
              ld_matrix = pair.right,
              output_prefix = out
      }
  }
  
  ```
  
  Since I'm grabbing the split files using glob in the previous step, I do not particularly know which element in the array corresponds to which chromosome. So I would be making a mistake if I zip the summary stats for one chromosome with the LD matrix corresponding to a different chromosome.
  
  The programmatic way would be to iterate the scatter over chromosome name but then I would lose the ability to specify the dependency between the two tasks.
  
  Is there a better way to do this?
  
  Cross posted on biostars (<https://www.biostars.org/p/434582/>) so pardon me if that is discouraged.
  
  **Tags:** wdl
  
  > **Comments:**
  > **username_1**: Two questions: 1. Would it be possible for `file_of_ld_matrices_by_chr` to be in JSON format, instead of just a list a list of file paths? 2. What does the name of the file produced by `splitGwas` look like (ie. what's in between the prefix and the `.assoc`?  
  > **username_0**: 1. Yes, I can put the LD matrices in JSON 2. The file names produced by split GWAS have the names ${prefix}_chromosome.assoc so if the user supplied prefix string is T2D as in type 2 diabetes, the outputs are T2D_1.assoc, T2D_2.assoc ... T2D_22.assoc  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-27. Score: 1 | Accepted answer
  
  You could use the [`sub`](https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#string-substring-string-string) function to obtain the chromosome from the globbed file names and then use that to retrieve the correct LD matrices from a Map (see [`read_map`](https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#mapstring-string-read_mapstringfile) or [`read_json`](https://github.com/openwdl/wdl/blob/master/versions/1.0/SPEC.md#mixed-read_jsonstringfile)).
  
  eg.:
  
  ```
  Map[String, String] ld_matrices = read_json(file_of_ld_matrices_by_chr)
  
  scatter(gwas_for_chr in split.gwas_by_chr) {
      String chr = sub(sub(gwas_for_chr, prefix, ""), "\.assoc$", "")
  
      call sbayes.run {
          input:
              GCTB = gctb_executable_path,
              gwas = gwas_for_chr,
              ld_matrix = ld_matrices[chr],
              output_prefix = out
      }
  }
  
  ```
  
  The `file_of_ld_matrices_by_chr` would have to look like this:
  
  ```
  {
    "chr1": "/path/to/file1",
    "chr2": "/path/to/file2"
  }
  
  ```
  
  > **Comments:**
  > **username_0**: This works perfectly.  
  
  ---
  
  