Answers.Body: <p>27GB is not huge. You can <code>zcat</code> the file and see the last few lines. It will take ~10 minutes. It is theoretically possible to peek a bgzf file, but no such tools are available.</p>
  
  <p>By the way,</p>
  
  <blockquote>
    <p>I am running bcftools merge to create a single .vcf.gz file out of a collection of chromosomal .vcf.gz files from the 1000g project.</p>
  </blockquote>
  
  <p>Probably you should use <code>bcftools concat</code>, not <code>bcftools merge</code>. <code>merge</code> combines VCFs from different samples. <code>concat</code> concatenates VCFs for individual chromosomes; each input VCF contains the same set of samples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T17:11:01.950Z
Answers.Text: Ah! I didn't know about `bcftools concat`, I will definitely try that. The `bcftools merge` has been running for several days and no idea what's taking that long.
Answers.UserDisplayName: 719016
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T13:26:31.247Z
Answers.LastActivityDate: 2019-06-28T19:05:02.320Z
Answers.LastEditDate: 2019-06-28T19:05:02.320Z
Answers.UserDisplayName: user172818
Answers.Body: <p>You could find which Chromosome the merged file is currently writing on by:</p>
  
  <pre><code>bcftools query -f'%CHROM\n' final.file.vcf.gz | tail
  </code></pre>
  
  <p>Note that also you might want to try multithreading the command that you are using to write the file, as bcftools is capable of using several threads for the compression of output files. E.g.:</p>
  
  <pre><code>bcftools merge -Ob --threads 32 file1.vcf.gz file2.vcf.gz &gt; mergedFile.vcf.gz 
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-02T00:44:18.747Z
Answers.LastActivityDate: 2019-07-02T00:44:18.747Z
Answers.UserDisplayName: user438383
Body: <p>I am running <code>bcftools merge</code> to create a single <code>.vcf.gz</code> file out of a collection of chromosomal <code>.vcf.gz</code> files from the 1000g project.</p>
  
  <p>The merge process has been running for a couple of days, and I want to estimate how much longer it will take. The file is a growing <code>.vcf.gz</code> that is being <code>bgzip -c</code> compressed from a Linux pipe:</p>
  
  <pre><code>bcftools merge [...] | some processing and filtering | bgzip -c &gt; final.file.vcf.gz
  </code></pre>
  
  <p>How can I know at which point is the file currently, to predict how much longer it will take to complete the merge?</p>
  
  <p>Is there a way to do something like:</p>
  
  <pre><code>tail --with-some-magic final.file.vcf.gz | bgzip -d -c
  </code></pre>
  
  <p>So that I can read at which chromosome is currently on?</p>
  
  <p>Is there a way to guesstimate the final size of the merge file and compare from the sum of sizes of the original files? E.g. the sum of compressed sizes originally (before filtering and munging) is 20G:</p>
  
  <pre><code>find <span class="math-container">$PWD -name "ALL.chr*.vcf.gz" | xargs ls -l | awk '{print $</span>5}' | csvtk summary -H -f 1:sum | numfmt --to-si
  </code></pre>
  
  <p>This gives 20G, but the merge file is at 27G and still running.</p>
  
  <p>Any ideas?</p>
  
Tags: <vcf><vcftools><bcftools>
Title: extracting last few lines of a growing bgzip compressed .vcf.gz file
ThreadText: # extracting last few lines of a growing bgzip compressed .vcf.gz file
  
  (Asked by: username_0 on 2019-06-27)
  
  I am running `bcftools merge` to create a single `.vcf.gz` file out of a collection of chromosomal `.vcf.gz` files from the 1000g project.
  
  The merge process has been running for a couple of days, and I want to estimate how much longer it will take. The file is a growing `.vcf.gz` that is being `bgzip -c` compressed from a Linux pipe:
  
  ```
  bcftools merge [...] | some processing and filtering | bgzip -c > final.file.vcf.gz
  
  ```
  
  How can I know at which point is the file currently, to predict how much longer it will take to complete the merge?
  
  Is there a way to do something like:
  
  ```
  tail --with-some-magic final.file.vcf.gz | bgzip -d -c
  
  ```
  
  So that I can read at which chromosome is currently on?
  
  Is there a way to guesstimate the final size of the merge file and compare from the sum of sizes of the original files? E.g. the sum of compressed sizes originally (before filtering and munging) is 20G:
  
  ```
  find $PWD -name "ALL.chr*.vcf.gz" | xargs ls -l | awk '{print $5}' | csvtk summary -H -f 1:sum | numfmt --to-si
  
  ```
  
  This gives 20G, but the merge file is at 27G and still running.
  
  Any ideas?
  
  **Tags:** vcf, vcftools, bcftools
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-06-28. Score: 5
  
  27GB is not huge. You can `zcat` the file and see the last few lines. It will take ~10 minutes. It is theoretically possible to peek a bgzf file, but no such tools are available.
  
  By the way,
  
  > I am running bcftools merge to create a single .vcf.gz file out of a collection of chromosomal .vcf.gz files from the 1000g project.
  
  Probably you should use `bcftools concat`, not `bcftools merge`. `merge` combines VCFs from different samples. `concat` concatenates VCFs for individual chromosomes; each input VCF contains the same set of samples.
  
  > **Comments:**
  > **username_0**: Ah! I didn't know about `bcftools concat`, I will definitely try that. The `bcftools merge` has been running for several days and no idea what's taking that long.  
  
  ---
  
  ### Answer by username_2 on 2019-07-02. Score: 0
  
  You could find which Chromosome the merged file is currently writing on by:
  
  ```
  bcftools query -f'%CHROM\n' final.file.vcf.gz | tail
  
  ```
  
  Note that also you might want to try multithreading the command that you are using to write the file, as bcftools is capable of using several threads for the compression of output files. E.g.:
  
  ```
  bcftools merge -Ob --threads 32 file1.vcf.gz file2.vcf.gz > mergedFile.vcf.gz 
  
  ```
  
  
  ---
  
  