Answers.Body: <p>We created this over the pandemic when we needed to merge loads of VCFs which all have the same header and first few columns (ie same variants)</p>
  <p><a href="https://github.com/iqbal-lab-org/ivcfmerge" rel="nofollow noreferrer">https://github.com/iqbal-lab-org/ivcfmerge</a></p>
  <p>That page gives detailed instructions but i'll paste a couple of examples
  of how you can use it</p>
  <p>== In Python
  =4.1.1 If the number of input files is small (can be opened all at once)</p>
  <pre><code>from contextlib import ExitStack
  from ivcfmerge import ivcfmerge
  
  filenames = [...]    # List/iterator of relative/absolute paths to input files
  output_path = '...'  # Where to write the merged VCF to
  
  with ExitStack() as stack:
      files = map(lambda fname: stack.enter_context(open(fname)), filenames)
      with open(output_path) as outfile:
          ivcfmerge(files, outfile)
  </code></pre>
  <p>=4.1.2 If the number of input files is big (cannot be opened all at once)
  from ivcfmerge import ivcfmerge_batch</p>
  <pre><code>filenames = [...]    # List/iterator of relative/absolute paths to input files
  output_path = '...'  # Where to write the merged VCF to
  batch_size = 1000    # How many files to open and merge at once
  
  ivcfmerge_batch(filenames, output_path, batch_size)
  </code></pre>
  <p>Pretty sure we initially did batched bcftools merge, I can't honestly remember why we moved to this. Thankfully not having to do these merges very often</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T22:50:42.480Z
Answers.Text: Thanks Zam. Unfortunately I fail the first two assumptions listed in the README.
Answers.UserDisplayName: Michael Hall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T08:04:23.903Z
Answers.LastActivityDate: 2023-04-18T08:16:02.730Z
Answers.LastEditDate: 2023-04-18T08:16:02.730Z
Answers.UserDisplayName: Zamin Iqbal
Answers.Body: <p>I ended up using an iterative approach (inspired by <a href="https://shicheng-guo.github.io/bioinformatics/1923/02/28/bcftools-merge" rel="nofollow noreferrer">this post</a>), which took 1.5 hours - so a massive speed up (and that was just the sequential version).</p>
  <p>Note: <a href="https://github.com/sharkdp/fd" rel="nofollow noreferrer"><code>fd</code></a> is analogous to <code>find</code>, but way nicer to use.</p>
  <p>First, merge subsets of <em>n</em> VCFs</p>
  <pre><code># size of the subsets
  n=200
  # create file of filenames for each subset
  fd -e vcf . path/to/vcfs/ | split -l $n --additional-suffix '.fofn' - subset_vcfs
  </code></pre>
  <p><code>-e vcf</code> tells <code>fd</code> to find all files with a <code>.vcf</code> extension, you can obviously change this to <code>.bcf</code>/<code>.vcf.gz</code> etc. if necessary. <code>split</code> splits the list of all VCFs files into subsets of size <code>-l</code> (200) and creates the file of filenames with the prefix <code>subset_vcfs</code>. It will add a random suffix to this - e.g. <code>subset_vcfsav</code>. Plus the additional suffix <code>.fofn</code>. See the <a href="https://www.man7.org/linux/man-pages/man1/split.1.html" rel="nofollow noreferrer"><code>split</code> manual</a> for all options.</p>
  <p>Now iterate through these subsets and merge the VCFs listed in them.</p>
  <h2>Subset merge option 1 - sequential</h2>
  <p>This just merges the VCFs one at a time</p>
  <pre class="lang-bash prettyprint-override"><code>for sub in *.fofn;
  do
      echo &quot;Merging subset ${sub}...&quot;
      out_vcf=&quot;merge.${sub}.bcf&quot;
      bcftools merge --file-list &quot;$sub&quot; -o &quot;$out_vcf&quot;
      bcftools index &quot;$out_vcf&quot;
  done
  </code></pre>
  <h2>Subset merge option 2 - parallel</h2>
  <p>You can easily parallelise the previous step using <code>fd</code> on the subset files. See <a href="https://mbh.sh/2020-06-22-cheap-parallelisation/" rel="nofollow noreferrer">this post</a> for more details. Or basic <a href="https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop">bash <code>for</code> loop parallelisation</a>.</p>
  <p>Create a script (<code>merge_subset.sh</code>) of what we want to run on each subset</p>
  <pre class="lang-bash prettyprint-override"><code>#!/usr/bin/env bash
  sub=&quot;$1&quot;
  echo &quot;Merging subset ${sub}...&quot;
  out_vcf=&quot;merge.${sub}.bcf&quot;
  bcftools merge --file-list &quot;$sub&quot; -o &quot;$out_vcf&quot;
  bcftools index &quot;$out_vcf&quot;
  </code></pre>
  <p>Now we run this script with each subset file of filenames using <code>fd</code></p>
  <pre><code># number of processes to use. remove this option to use all available
  jobs=4
  fd -j $jobs -e fofn -x bash merge_subset.sh '{}'
  </code></pre>
  <h2>Final merge</h2>
  <p>Now we merge all of these VCFs into our final VCF</p>
  <pre><code>ls merge.*.bcf &gt; merged.fofn
  bcftools merge -o merged.bcf --file-list merged.fofn
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-19T01:52:02.690Z
Answers.LastActivityDate: 2023-04-19T23:32:39.223Z
Answers.LastEditDate: 2023-04-19T23:32:39.223Z
Answers.UserDisplayName: Michael Hall
Answers.Body: <p>Perform a tree based merge of the files.</p>
  <ul>
  <li>Sort the files into groups that can easily be merged in a measurable amount of time.</li>
  <li>Merge each group into a separate Variant Call Format (VCF) file
  <ul>
  <li>Has the benefit that files can also be grouped by type / source / run</li>
  <li>Further benefit that small merges can be checked to ensure you like the output before proceeding to the next stage.</li>
  <li>Further benefit that there are also subgroup files available to be distributed or shared separately.</li>
  <li>Downside, extra storage space until you have a final product.</li>
  </ul>
  </li>
  <li>Merge subfiles upward in stages until you arrive at a complete 44,000 entry file
  <ul>
  <li>If you're happy with the earlier test results, process can be automated with relatively simple iterative shell scripts.</li>
  <li>Possibly faster, as the bulk merge needs to load every single file into memory, and you may be hitting a capacity bottleneck.</li>
  <li>Worked in Fluid Dynamics for years, and often ran into a similar issue.  Could run many small cases representing a large case faster than running the large case, because large case would overwhelm memory and transfer capacity.</li>
  <li>In some clusters you may also be causing nodes to silently crash, which then stalls the entire process without a visible notification.</li>
  </ul>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T17:55:31.967Z
Answers.LastActivityDate: 2023-04-18T17:55:31.967Z
Answers.UserDisplayName: G. Putnam
Body: <p>Can anyone suggest a strategy for speeding up VCF merging?</p>
  <p>I have ~44,000 single-sample VCFs that I am trying to merge into a multi-sample VCF with <code>bcftools merge</code>, but the job keeps timing out on the cluster (just timed out after 4 days). I could obviously try running it for a couple of weeks, but I was hoping there is a better strategy someone can suggest?</p>
  <p>The exact command I am running is</p>
  <pre><code>bcftools merge --no-version -m all -i 'DP:avg,RO:avg,AO:avg' -F + -o merged.vcf --file-list merge.fofn
  </code></pre>
  <p>Happy to try any other tool that will give the same merge results.</p>
  
Tags: <vcf><bcftools><merge>
Title: Strategy for merging many VCFs
ThreadText: # Strategy for merging many VCFs
  
  (Asked by: username_0 on 2023-04-18)
  
  Can anyone suggest a strategy for speeding up VCF merging?
  
  I have ~44,000 single-sample VCFs that I am trying to merge into a multi-sample VCF with `bcftools merge`, but the job keeps timing out on the cluster (just timed out after 4 days). I could obviously try running it for a couple of weeks, but I was hoping there is a better strategy someone can suggest?
  
  The exact command I am running is
  
  ```
  bcftools merge --no-version -m all -i 'DP:avg,RO:avg,AO:avg' -F + -o merged.vcf --file-list merge.fofn
  
  ```
  
  Happy to try any other tool that will give the same merge results.
  
  **Tags:** vcf, bcftools, merge
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-04-18. Score: 4
  
  We created this over the pandemic when we needed to merge loads of VCFs which all have the same header and first few columns (ie same variants)
  
  <https://github.com/username_1-lab-org/ivcfmerge>
  
  That page gives detailed instructions but i'll paste a couple of examples
  of how you can use it
  
  == In Python
  =4.1.1 If the number of input files is small (can be opened all at once)
  
  ```
  from contextlib import ExitStack
  from ivcfmerge import ivcfmerge
  
  filenames = [...]    # List/iterator of relative/absolute paths to input files
  output_path = '...'  # Where to write the merged VCF to
  
  with ExitStack() as stack:
      files = map(lambda fname: stack.enter_context(open(fname)), filenames)
      with open(output_path) as outfile:
          ivcfmerge(files, outfile)
  
  ```
  
  =4.1.2 If the number of input files is big (cannot be opened all at once)
  from ivcfmerge import ivcfmerge\_batch
  
  ```
  filenames = [...]    # List/iterator of relative/absolute paths to input files
  output_path = '...'  # Where to write the merged VCF to
  batch_size = 1000    # How many files to open and merge at once
  
  ivcfmerge_batch(filenames, output_path, batch_size)
  
  ```
  
  Pretty sure we initially did batched bcftools merge, I can't honestly remember why we moved to this. Thankfully not having to do these merges very often
  
  > **Comments:**
  > **username_0**: Thanks Zam. Unfortunately I fail the first two assumptions listed in the README.  
  
  ---
  
  ### Answer by username_0 on 2023-04-19. Score: 3
  
  I ended up using an iterative approach (inspired by [this post](https://shicheng-guo.github.io/bioinformatics/1923/02/28/bcftools-merge)), which took 1.5 hours - so a massive speed up (and that was just the sequential version).
  
  Note: [`fd`](https://github.com/sharkdp/fd) is analogous to `find`, but way nicer to use.
  
  First, merge subsets of *n* VCFs
  
  ```
  # size of the subsets
  n=200
  # create file of filenames for each subset
  fd -e vcf . path/to/vcfs/ | split -l $n --additional-suffix '.fofn' - subset_vcfs
  
  ```
  
  `-e vcf` tells `fd` to find all files with a `.vcf` extension, you can obviously change this to `.bcf`/`.vcf.gz` etc. if necessary. `split` splits the list of all VCFs files into subsets of size `-l` (200) and creates the file of filenames with the prefix `subset_vcfs`. It will add a random suffix to this - e.g. `subset_vcfsav`. Plus the additional suffix `.fofn`. See the [`split` manual](https://www.man7.org/linux/man-pages/man1/split.1.html) for all options.
  
  Now iterate through these subsets and merge the VCFs listed in them.
  
  ## Subset merge option 1 - sequential
  
  This just merges the VCFs one at a time
  
  ```bash
  for sub in *.fofn;
  do
      echo "Merging subset ${sub}..."
      out_vcf="merge.${sub}.bcf"
      bcftools merge --file-list "$sub" -o "$out_vcf"
      bcftools index "$out_vcf"
  done
  
  ```
  
  ## Subset merge option 2 - parallel
  
  You can easily parallelise the previous step using `fd` on the subset files. See [this post](https://mbh.sh/2020-06-22-cheap-parallelisation/) for more details. Or basic [bash `for` loop parallelisation](https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop).
  
  Create a script (`merge_subset.sh`) of what we want to run on each subset
  
  ```bash
  #!/usr/bin/env bash
  sub="$1"
  echo "Merging subset ${sub}..."
  out_vcf="merge.${sub}.bcf"
  bcftools merge --file-list "$sub" -o "$out_vcf"
  bcftools index "$out_vcf"
  
  ```
  
  Now we run this script with each subset file of filenames using `fd`
  
  ```
  # number of processes to use. remove this option to use all available
  jobs=4
  fd -j $jobs -e fofn -x bash merge_subset.sh '{}'
  
  ```
  
  ## Final merge
  
  Now we merge all of these VCFs into our final VCF
  
  ```
  ls merge.*.bcf > merged.fofn
  bcftools merge -o merged.bcf --file-list merged.fofn
  
  ```
  
  
  ---
  
  ### Answer by username_2 on 2023-04-18. Score: 1
  
  Perform a tree based merge of the files.
  
  * Sort the files into groups that can easily be merged in a measurable amount of time.
  * Merge each group into a separate Variant Call Format (VCF) file
    + Has the benefit that files can also be grouped by type / source / run
    + Further benefit that small merges can be checked to ensure you like the output before proceeding to the next stage.
    + Further benefit that there are also subgroup files available to be distributed or shared separately.
    + Downside, extra storage space until you have a final product.
  * Merge subfiles upward in stages until you arrive at a complete 44,000 entry file
    + If you're happy with the earlier test results, process can be automated with relatively simple iterative shell scripts.
    + Possibly faster, as the bulk merge needs to load every single file into memory, and you may be hitting a capacity bottleneck.
    + Worked in Fluid Dynamics for years, and often ran into a similar issue. Could run many small cases representing a large case faster than running the large case, because large case would overwhelm memory and transfer capacity.
    + In some clusters you may also be causing nodes to silently crash, which then stalls the entire process without a visible notification.
  
  
  ---
  
  