Answers.Body: <p>The best tool for the job is probably <code>bcftools</code> as <a href="https://bioinformatics.stackexchange.com/a/3306/298">suggested</a> by Bioathlete, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:</p>
  
  <pre><code>grep '^##' chr*vcf | sort | uniq &gt; all.vcf
  grep -m1 '^#CHR' chr1.vcf &gt;&gt; all.vcf  ## Get the chr header line 
  grep -v '^#' chr*vcf &gt;&gt; all.vcf
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T01:57:16.013Z
Answers.Text: I appear to get a malformed VCF header with the above commands
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T08:26:10.147Z
Answers.Text: @ShanZhengYang malformed how? Could you edit your question and show us an example of your files?
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T18:32:14.970Z
Answers.Text: So, I see the above `grep` commands give the same behavior with all of my standard VCFs. I'll get something like this in `all.vcf`:
  `chr10.vcf:##contig=<ID=chr10,length=133797422>
  chr10.vcf:##fileDate=20121011
  chr10.vcf:##fileformat=VCFv4.2
  chr10.vcf:##reference=path/reference.fa
  chr10.vcf:##source=foobar
  chr11.vcf:##contig=<ID=chr10,length=133797422>
  chr11.vcf:##fileDate=20121011
  chr11.vcf:##fileformat=VCFv4.2
  chr11.vcf:##reference=path/reference.fa
  chr11.vcf:##source=foobar
  ....`
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-15T15:13:06.070Z
Answers.LastActivityDate: 2018-01-15T15:13:06.070Z
Answers.UserDisplayName: terdon
Answers.Body: <p>use picard <strong>GatherVcf</strong>: <a href="http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs" rel="nofollow noreferrer">http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs</a></p>
  
  <blockquote>
    <p>Gathers multiple VCF files from a scatter operation into a single VCF file. Input files must be supplied in genomic order and must not have events at overlapping positions.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T01:53:58.560Z
Answers.Text: The error I get here is `Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.`. It is true, my "by chromosome"-VCFs do not contain a `##contig` line in the header...but I'm not sure that would matter
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T02:06:14.400Z
Answers.Text: Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326 
  Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T16:38:38.303Z
Answers.Text: "but I'm not sure that would matter" it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary https://broadinstitute.github.io/picard/command-line-overview.html
Answers.UserDisplayName: Pierre
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T18:20:32.423Z
Answers.Text: "it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary". Ah, I understand now. Thanks for the help!
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-15T18:30:09.480Z
Answers.LastActivityDate: 2018-01-15T18:30:09.480Z
Answers.UserDisplayName: Pierre
Answers.Body: <p>I would recommend <a href="https://samtools.github.io/bcftools/bcftools.html" rel="noreferrer">bcftools</a> concat.  You can't just <code>cat</code> them together because each file has a header section.  The bcftools command will handle all that for you.  Each vcf file must be sorted prior to calling <code>concat</code></p>
  
  <p><code>bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf</code></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-15T03:27:21.287Z
Answers.Text: It's a bit odd. `vcf-validator chr1.vcf` shows no problems. I believe the output of `bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf` shows an error: `FIXME: sequence name chr1 in chr1.vcf`
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-15T15:09:32.983Z
Answers.Text: @ShanZhengYang the errors seem to indicate that you have `chr1` in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the `chr1.vcf` file? Maybe the validator somehow is missing this error?
Answers.UserDisplayName: juod
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-15T19:06:45.167Z
Answers.Text: @juod The VCF header looks correct to me. `#CHROM POS ID REF ALT QUAL FILTER INFO`
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-15T23:41:00.307Z
Answers.Text: @juod "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T00:58:10.817Z
Answers.Text: According to the VCF spec https://samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - `##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x>`
Answers.UserDisplayName: Bioathlete
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T02:05:19.847Z
Answers.Text: @Bioathlete I see. I think I understand now---this issue has been discussed before: https://github.com/samtools/bcftools/issues/326
  I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T09:34:19.263Z
Answers.Text: @ShanZhengYang try that - it should work. I don't even think it's necessary to provide the length, but you could experiment and let us know.
Answers.UserDisplayName: juod
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-16T18:20:12.587Z
Answers.Text: @juod The above worked! Thanks for the help!
Answers.UserDisplayName: ShanZhengYang
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-14T23:25:10.867Z
Answers.LastActivityDate: 2018-01-14T23:25:10.867Z
Answers.UserDisplayName: Bioathlete
Body: <p>I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc. </p>
  
  <p>I checked to make sure that these VCFs were valid via <code>VCFtools</code>, i.e. </p>
  
  <pre><code>$ vcf-validator chr1.vcf
  </code></pre>
  
  <p>which works---these are valid VCFs I was given. </p>
  
  <p>Now, I would like to combine these VCFs into one VCF. </p>
  
  <p>I naïvely tried the following <code>cat</code> operation:</p>
  
  <pre><code>$ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf &gt; total_chroms.vcf
  </code></pre>
  
  <p>This doesn't work properly though</p>
  
  <pre><code>$ vcf-validator total_chroms.vcf
  The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.)
  Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n]
   at /path/vcftools-0.1.14/perl/Vcf.pm line 172, &lt;__ANONIO__&gt; line 191016.
       Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 335
       Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 3457
       Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 2574
       VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 60
       main::do_validation('HASH(0x16ada68)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 14
  $
  </code></pre>
  
  <p>What tools are available to merge these VCF's together into a total VCF? </p>
  
Tags: <vcf><vcftools>
Title: How to concatenate "by chromosome"-VCFs?
ThreadText: # How to concatenate "by chromosome"-VCFs?
  
  (Asked by: username_0 on 2018-01-14)
  
  I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc.
  
  I checked to make sure that these VCFs were valid via `VCFtools`, i.e.
  
  ```
  $ vcf-validator chr1.vcf
  
  ```
  
  which works---these are valid VCFs I was given.
  
  Now, I would like to combine these VCFs into one VCF.
  
  I naïvely tried the following `cat` operation:
  
  ```
  $ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf > total_chroms.vcf
  
  ```
  
  This doesn't work properly though
  
  ```
  $ vcf-validator total_chroms.vcf
  The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.)
  Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n]
   at /path/vcftools-0.1.14/perl/Vcf.pm line 172, <__ANONIO__> line 191016.
       Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 335
       Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 3457
       Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 2574
       VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 60
       main::do_validation('HASH(0x16ada68)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 14
  $
  
  ```
  
  What tools are available to merge these VCF's together into a total VCF?
  
  **Tags:** vcf, vcftools
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2018-01-14. Score: 12 | Accepted answer
  
  I would recommend [bcftools](https://samtools.github.io/bcftools/bcftools.html) concat. You can't just `cat` them together because each file has a header section. The bcftools command will handle all that for you. Each vcf file must be sorted prior to calling `concat`
  
  `bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf`
  
  > **Comments:**
  > **username_0**: It's a bit odd. `vcf-validator chr1.vcf` shows no problems. I believe the output of `bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf` shows an error: `FIXME: sequence name chr1 in chr1.vcf`  
  > **username_4**: @username_0 the errors seem to indicate that you have `chr1` in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the `chr1.vcf` file? Maybe the validator somehow is missing this error?  
  > **username_0**: @username_4 The VCF header looks correct to me. `#CHROM POS ID REF ALT QUAL FILTER INFO`  
  > **username_0**: @username_4 "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header  
  > **username_3**: According to the VCF spec https://samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - `##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x>`  
  > **username_0**: @username_3 I see. I think I understand now---this issue has been discussed before: https://github.com/samtools/bcftools/issues/326 I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38  
  > **username_4**: @username_0 try that - it should work. I don't even think it's necessary to provide the length, but you could experiment and let us know.  
  > **username_0**: @username_4 The above worked! Thanks for the help!  
  
  ---
  
  ### Answer by username_1 on 2018-01-15. Score: 2
  
  The best tool for the job is probably `bcftools` as [suggested](https://bioinformatics.stackexchange.com/a/3306/298) by username_3, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:
  
  ```
  grep '^##' chr*vcf | sort | uniq > all.vcf
  grep -m1 '^#CHR' chr1.vcf >> all.vcf  ## Get the chr header line 
  grep -v '^#' chr*vcf >> all.vcf
  
  ```
  
  > **Comments:**
  > **username_0**: I appear to get a malformed VCF header with the above commands  
  > **username_1**: @username_0 malformed how? Could you edit your question and show us an example of your files?  
  > **username_0**: So, I see the above `grep` commands give the same behavior with all of my standard VCFs. I'll get something like this in `all.vcf`: `chr10.vcf:##contig=<ID=chr10,length=133797422> chr10.vcf:##fileDate=20121011 chr10.vcf:##fileformat=VCFv4.2 chr10.vcf:##reference=path/reference.fa chr10.vcf:##source=foobar chr11.vcf:##contig=<ID=chr10,length=133797422> chr11.vcf:##fileDate=20121011 chr11.vcf:##fileformat=VCFv4.2 chr11.vcf:##reference=path/reference.fa chr11.vcf:##source=foobar ....`  
  
  ---
  
  ### Answer by username_2 on 2018-01-15. Score: 2
  
  use picard **GatherVcf**: <http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs>
  
  > Gathers multiple VCF files from a scatter operation into a single VCF file. Input files must be supplied in genomic order and must not have events at overlapping positions.
  
  > **Comments:**
  > **username_0**: The error I get here is `Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.`. It is true, my "by chromosome"-VCFs do not contain a `##contig` line in the header...but I'm not sure that would matter  
  > **username_0**: Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326  Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38  
  > **username_2**: "but I'm not sure that would matter" it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary https://broadinstitute.github.io/picard/command-line-overview.html  
  > **username_0**: "it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary". Ah, I understand now. Thanks for the help!  
  
  ---
  
  