Answers.Body: <p>You should go with the CSI index as it seems like the BAI indexes cannot handle chromosomes longer than 2^29.</p>
  
  <p>Your problem is that you're trying to merge the index files using <code>vcf-merge</code>. You should merge the VCF files, not the <code>.csi</code> index files.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T08:55:30.327Z
Answers.Text: Do you have a reference for this limitation? Is it mentioned in the docs somewhere?
Answers.UserDisplayName: terdon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T17:39:49.787Z
Answers.Text: Yeah, I found it in a GitHub issue for the mosdepth/samtools repos: https://github.com/brentp/mosdepth/issues/41 https://github.com/samtools/samtools/issues/241
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T18:14:02.323Z
Answers.Text: This has come up on the mailing lists and forums often enough over the years to be part of the common lore. It's spelt out in the BAI spec in [SAMv1.pdf](http://samtools.github.io/hts-specs/SAMv1.pdf) §5.1.1 and will soon be mentioned in the `samtools index` manual page ([PR #1248](https://github.com/samtools/samtools/pull/1248)).
Answers.UserDisplayName: John Marshall
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T23:59:45.620Z
Answers.Text: Thanks for the advice, and sorry for the cross-post. You're right, CSI must be used if indexing is necessary since all of my chromosomes are larger than 2^29.
Answers.UserDisplayName: Brian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T23:08:08.410Z
Answers.LastActivityDate: 2020-05-13T23:08:08.410Z
Answers.UserDisplayName: Ram RS
Body: <p>I'm trying to use vcf-merge to combine 2 exome capture vcf files (~250K and ~330K in size) before trying it on all 96 samples. I'd appreciate any advice on the best way to do that! I've detailed what I've tried below. My issue seems to be with using tabix to convert the files to .tbi format.</p>
  <p>Step 1: BGZIP</p>
  <p>So far, I've zipped the files without issue:</p>
  <pre><code>bgzip sample1.vcf
  bgzip sample2.vcf
  </code></pre>
  <p>which produces:</p>
  <pre><code>sample1.vcf.gz
  sample2.vcf.gz
  </code></pre>
  <p>Step 2: TABIX</p>
  <p>When I try to use this command:</p>
  <pre><code>tabix -h -p vcf sample1.vcf.gz
  </code></pre>
  <p>the stderr is:</p>
  <blockquote>
  <p>Region 536999277..536999278 cannot be stored in a tbi index. Try using
  a csi index with min_shift = 14, n_lvls &gt;= 6. tbx_index_build failed:
  sample1.vcf.gz</p>
  </blockquote>
  <p>Using the -C option which works:</p>
  <pre><code>tabix -C -h -p vcf sample1.vcf.gz
  tabix -C -h -p vcf sample2.vcf.gz
  </code></pre>
  <p>which produces:
  sample1.vcf.gz.csi
  sample2.vcf.gz.csi</p>
  <p>Step 3: VCF-MERGE</p>
  <p>When I use this command:</p>
  <pre><code>vcf-merge sample1.vcf.gz.csi sample2.vcf.gz.csi &gt; out.vcf.gz
  </code></pre>
  <p>the merge fails, and I get this stderr:</p>
  <blockquote>
  <p>Broken VCF header, no column names? at /usr/share/perl5/Vcf.pm line
  172, &lt;<strong>ANONIO</strong>&gt; line 1. Vcf::throw(Vcf4_2=HASH(0x5645a760bc38),
  &quot;Broken VCF header, no column names?&quot;) called at
  /usr/share/perl5/Vcf.pm line 867
  VcfReader::_read_column_names(Vcf4_2=HASH(0x5645a760bc38)) called at
  /usr/share/perl5/Vcf.pm line 602
  VcfReader::parse_header(Vcf4_2=HASH(0x5645a760bc38)) called at
  /usr/bin/vcf-merge line 183 main::init_cols(HASH(0x5645a761f438),
  Vcf4_2=HASH(0x5645a760b248)) called at /usr/bin/vcf-merge line 279
  main::merge_vcf_files(HASH(0x5645a761f438)) called at
  /usr/bin/vcf-merge line 12</p>
  </blockquote>
  <p>If I exclude the tabix files, the merge still fails and the stderr says:</p>
  <blockquote>
  <p>The column names not tab-separated? Could not load .tbi index of
  sample1.vcf.gz. The command exited with an error. Is the file tabix
  indexed?</p>
  </blockquote>
  <p>Any suggestions on how I can correctly merge these files?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-13T23:28:03.333Z
Comments.Text: Cross-posted on Biostars: https://www.biostars.org/p/437957/
Comments.UserDisplayName: Ram RS
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-14T08:55:57.097Z
Comments.Text: Would it be possible to share the files with us? Do you really have a chromosome that's 536999278 bases long? What species are you working on?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-14T23:55:44.067Z
Comments.Text: Hi, I'm working with conifers which have very long chromosomes. As discusses below, the tbi cutoff is ~536,870,912 bp, so csi is needed.
Comments.UserDisplayName: Brian
Tags: <vcf><linux><vcftools><tabix>
Title: VCF-merge fails due to tabix not producing .tbi files
ThreadText: # VCF-merge fails due to tabix not producing .tbi files
  
  (Asked by: username_0 on 2020-05-13)
  
  I'm trying to use vcf-merge to combine 2 exome capture vcf files (~250K and ~330K in size) before trying it on all 96 samples. I'd appreciate any advice on the best way to do that! I've detailed what I've tried below. My issue seems to be with using tabix to convert the files to .tbi format.
  
  Step 1: BGZIP
  
  So far, I've zipped the files without issue:
  
  ```
  bgzip sample1.vcf
  bgzip sample2.vcf
  
  ```
  
  which produces:
  
  ```
  sample1.vcf.gz
  sample2.vcf.gz
  
  ```
  
  Step 2: TABIX
  
  When I try to use this command:
  
  ```
  tabix -h -p vcf sample1.vcf.gz
  
  ```
  
  the stderr is:
  
  > Region 536999277..536999278 cannot be stored in a tbi index. Try using
  > a csi index with min\_shift = 14, n\_lvls >= 6. tbx\_index\_build failed:
  > sample1.vcf.gz
  
  Using the -C option which works:
  
  ```
  tabix -C -h -p vcf sample1.vcf.gz
  tabix -C -h -p vcf sample2.vcf.gz
  
  ```
  
  which produces:
  sample1.vcf.gz.csi
  sample2.vcf.gz.csi
  
  Step 3: VCF-MERGE
  
  When I use this command:
  
  ```
  vcf-merge sample1.vcf.gz.csi sample2.vcf.gz.csi > out.vcf.gz
  
  ```
  
  the merge fails, and I get this stderr:
  
  > Broken VCF header, no column names? at /usr/share/perl5/Vcf.pm line
  > 172, <**ANONIO**> line 1. Vcf::throw(Vcf4\_2=HASH(0x5645a760bc38),
  > "Broken VCF header, no column names?") called at
  > /usr/share/perl5/Vcf.pm line 867
  > VcfReader::\_read\_column\_names(Vcf4\_2=HASH(0x5645a760bc38)) called at
  > /usr/share/perl5/Vcf.pm line 602
  > VcfReader::parse\_header(Vcf4\_2=HASH(0x5645a760bc38)) called at
  > /usr/bin/vcf-merge line 183 main::init\_cols(HASH(0x5645a761f438),
  > Vcf4\_2=HASH(0x5645a760b248)) called at /usr/bin/vcf-merge line 279
  > main::merge\_vcf\_files(HASH(0x5645a761f438)) called at
  > /usr/bin/vcf-merge line 12
  
  If I exclude the tabix files, the merge still fails and the stderr says:
  
  > The column names not tab-separated? Could not load .tbi index of
  > sample1.vcf.gz. The command exited with an error. Is the file tabix
  > indexed?
  
  Any suggestions on how I can correctly merge these files?
  
  **Tags:** vcf, linux, vcftools, tabix
  
  > **Comments:**
  > **username_1**: Cross-posted on Biostars: https://www.biostars.org/p/437957/  
  > **username_2**: Would it be possible to share the files with us? Do you really have a chromosome that's 536999278 bases long? What species are you working on?  
  > **username_0**: Hi, I'm working with conifers which have very long chromosomes. As discusses below, the tbi cutoff is ~536,870,912 bp, so csi is needed.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-13. Score: 2
  
  You should go with the CSI index as it seems like the BAI indexes cannot handle chromosomes longer than 2^29.
  
  Your problem is that you're trying to merge the index files using `vcf-merge`. You should merge the VCF files, not the `.csi` index files.
  
  > **Comments:**
  > **username_2**: Do you have a reference for this limitation? Is it mentioned in the docs somewhere?  
  > **username_1**: Yeah, I found it in a GitHub issue for the mosdepth/samtools repos: https://github.com/brentp/mosdepth/issues/41 https://github.com/samtools/samtools/issues/241  
  > **username_3**: This has come up on the mailing lists and forums often enough over the years to be part of the common lore. It's spelt out in the BAI spec in [SAMv1.pdf](http://samtools.github.io/hts-specs/SAMv1.pdf) §5.1.1 and will soon be mentioned in the `samtools index` manual page ([PR #1248](https://github.com/samtools/samtools/pull/1248)).  
  > **username_0**: Thanks for the advice, and sorry for the cross-post. You're right, CSI must be used if indexing is necessary since all of my chromosomes are larger than 2^29.  
  
  ---
  
  