Answers.Body: <p><a href="https://bioinformatics.stackexchange.com/a/8599/96">John Marshall said</a>:</p>
  
  <blockquote>
    <p><em>Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.</em></p>
  </blockquote>
  
  <p>I just published <a href="https://github.com/bioforensics/rsidx" rel="nofollow noreferrer">rsidx</a>, a new package for indexing VCF files by rsID. As John suggests, I simply created a mapping from rsID to genomic coordinates. These are stored in an sqlite3 database.</p>
  
  <p>It currently takes...several...hours to index build 151 of dbSNP on GRCh38. I'm looking into optimizing sqlite3 parameters to accelerate the indexing, as well as an alternative approach based on a two step process (populate an in-memory sqlite3 db, then dump DB to disk). But in the meantime, the search capability works great.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-10T19:26:37.633Z
Answers.LastActivityDate: 2019-05-10T19:26:37.633Z
Answers.UserDisplayName: Daniel Standage
Answers.Body: <p><strong>high-perf-bio.</strong></p>
  <p>I present my <a href="https://github.com/PlatonB/high-perf-bio" rel="nofollow noreferrer"><em>high-perf-bio</em></a> project, which, for the most part, solves the problem of fast extraction of anything values from VCF, BED and unformatted data.</p>
  <p><strong>Preparation.</strong></p>
  <p>1/ Run the automatic downloading of the toolkit and installation of the dependencies.</p>
  <pre><code>wget https://github.com/PlatonB/high-perf-bio/archive/refs/heads/master.zip &amp;&amp; unzip -o master.zip &amp;&amp; rm master.zip &amp;&amp; cd high-perf-bio-master &amp;&amp; bash __install_3rd_party.sh
  </code></pre>
  <p>2/ Reboot.</p>
  <p>3/ <code>cd path/to/high-perf-bio-master</code></p>
  <p><strong>Create DB with rsIDs.</strong></p>
  <pre><code>python create.py -S path/to/gzipped_tables_dir -i rsIDs_field_path
  </code></pre>
  <p><strong>Search rsIDs set in DB.</strong></p>
  <pre><code>python annotate.py -S path/to/annotated_gzipped_tables_dir -D db_name -T path/to/trg_dir #-c rsIDs_column_num -f rsIDs_field_path
  </code></pre>
  <p>P.S.1. Detailed help for each <em>high-perf-bio</em> component: <code>python tool_name.py -h</code></p>
  <p>P.S.2. For VCF and BED you can specify the minimum of arguments.</p>
  <p>P.S.3. Readme in English is in the early stages of writing. If you have any questions, write to <a href="https://github.com/PlatonB/high-perf-bio/issues" rel="nofollow noreferrer">Issues</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-24T15:14:58.423Z
Answers.Text: This is not too different from Daniel's project - he is more credible on the site plus has written on the  performance of his tool. Adding that information would help us compare your program to his at a glance.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-09T19:57:54.450Z
Answers.Text: @RamRS
  
  "plus has written on the performance of his tool." The performance of such tools directly depends on the performance of the underlying DBMS. I don't think the speed difference between MongoDB and SQLite is significant for indexed data. "This is not too different from Daniel's project" My toolkit has more functionality. It contains 10 high-level command utilities for solving typical bioinformatics data retrieval tasks. Searching rsIDs is not the only feature of high-perf-bio.
Answers.UserDisplayName: PlatonB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T18:46:20.673Z
Answers.Text: Sure, your tool might be a jack of all trades but for the task at hand, is your tool faster - that is, for the same dataset, does MongoDB give you a significant advantage? In any case, my suggestion is based on the fact that you're not as well knows as Daniel, so providing at least as much info as he does will only help your case.
Answers.UserDisplayName: Ram RS
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-22T16:18:42.350Z
Answers.LastActivityDate: 2023-03-02T15:01:47.590Z
Answers.LastEditDate: 2023-03-02T15:01:47.590Z
Answers.UserDisplayName: PlatonB
Answers.Body: <p>I am not aware of any widespread index formats for query-by-QNAME for BAM or query-by-ID for VCF. I thought there were historical <em>samtools{-devel,-help}</em> threads to the effect of “in principle one could index a named-sorted BAM file, but there's insufficient demand so no-one's ever implemented it”, but I can't find any just now.</p>
  
  <p>Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.</p>
  
  <p>If you are using dbSNP, probably you have a dbSNP database or API that you can use to query by rsID and get genomic position back. So you can retrieve your VCF contents by rsID in two stages: do that dbSNP query, and use its results to find the variants in your VCF file by position.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-08T09:41:26.713Z
Answers.LastActivityDate: 2019-05-08T09:41:26.713Z
Answers.UserDisplayName: John Marshall
Answers.Body: <p>Take a look at <a href="https://github.com/arq5x/grabix" rel="nofollow noreferrer"><code>grabix</code></a>. Using <code>grabix</code> you could implement a binary search on a VCF sorted by rsID and compressed using <code>bgzip</code>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-08T13:04:33.220Z
Answers.LastActivityDate: 2019-05-08T13:04:33.220Z
Answers.UserDisplayName: winni2k
Body: <p>With tabix one can index a VCF file for rapid variant retrieval based on genomic position. I'm wondering if there are any tools that will index a VCF file to allow rapid retrieval using rsIDs and/or other metadata? I'm aware of awk/grep/vcftools one-liners for this purpose, but I'd like to avoid scanning a huge VCF each time I need to retrieve the coordinates of a new batch of rsIDs.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-02T15:37:55.770Z
Comments.Text: Is re-sorting the VCF files feasible? It's possible to use Tabix on generic tab-delimited files, so sorting and indexing by the RSID column might work.
Comments.UserDisplayName: afaulconbridge
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-21T11:46:38.873Z
Comments.Text: I attempted this, but it failed on most large VCFs I tried.
Comments.UserDisplayName: Daniel Standage
Tags: <vcf><file-formats><variation><indexing>
Title: Index VCF file for rapid access by rsID?
ThreadText: # Index VCF file for rapid access by rsID?
  
  (Asked by: username_0 on 2019-05-06)
  
  With tabix one can index a VCF file for rapid variant retrieval based on genomic position. I'm wondering if there are any tools that will index a VCF file to allow rapid retrieval using rsIDs and/or other metadata? I'm aware of awk/grep/vcftools one-liners for this purpose, but I'd like to avoid scanning a huge VCF each time I need to retrieve the coordinates of a new batch of rsIDs.
  
  **Tags:** vcf, file-formats, variation, indexing
  
  > **Comments:**
  > **username_1**: Is re-sorting the VCF files feasible? It's possible to use Tabix on generic tab-delimited files, so sorting and indexing by the RSID column might work.  
  > **username_0**: I attempted this, but it failed on most large VCFs I tried.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_0 on 2019-05-10. Score: 0 | Accepted answer
  
  [username_4 username_4 said](https://bioinformatics.stackexchange.com/a/8599/96):
  
  > *Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.*
  
  I just published [rsidx](https://github.com/bioforensics/rsidx), a new package for indexing VCF files by rsID. As username_4 suggests, I simply created a mapping from rsID to genomic coordinates. These are stored in an sqlite3 database.
  
  It currently takes...several...hours to index build 151 of dbSNP on GRCh38. I'm looking into optimizing sqlite3 parameters to accelerate the indexing, as well as an alternative approach based on a two step process (populate an in-memory sqlite3 db, then dump DB to disk). But in the meantime, the search capability works great.
  
  
  ---
  
  ### Answer by username_2 on 2022-02-22. Score: 2
  
  **high-perf-bio.**
  
  I present my [*high-perf-bio*](https://github.com/username_2/high-perf-bio) project, which, for the most part, solves the problem of fast extraction of anything values from VCF, BED and unformatted data.
  
  **Preparation.**
  
  1/ Run the automatic downloading of the toolkit and installation of the dependencies.
  
  ```
  wget https://github.com/username_2/high-perf-bio/archive/refs/heads/master.zip && unzip -o master.zip && rm master.zip && cd high-perf-bio-master && bash __install_3rd_party.sh
  
  ```
  
  2/ Reboot.
  
  3/ `cd path/to/high-perf-bio-master`
  
  **Create DB with rsIDs.**
  
  ```
  python create.py -S path/to/gzipped_tables_dir -i rsIDs_field_path
  
  ```
  
  **Search rsIDs set in DB.**
  
  ```
  python annotate.py -S path/to/annotated_gzipped_tables_dir -D db_name -T path/to/trg_dir #-c rsIDs_column_num -f rsIDs_field_path
  
  ```
  
  P.S.1. Detailed help for each *high-perf-bio* component: `python tool_name.py -h`
  
  P.S.2. For VCF and BED you can specify the minimum of arguments.
  
  P.S.3. Readme in English is in the early stages of writing. If you have any questions, write to [Issues](https://github.com/username_2/high-perf-bio/issues).
  
  > **Comments:**
  > **username_3**: This is not too different from username_0's project - he is more credible on the site plus has written on the  performance of his tool. Adding that information would help us compare your program to his at a glance.  
  > **username_2**: @username_3  "plus has written on the performance of his tool." The performance of such tools directly depends on the performance of the underlying DBMS. I don't think the speed difference between MongoDB and SQLite is significant for indexed data. "This is not too different from username_0's project" My toolkit has more functionality. It contains 10 high-level command utilities for solving typical bioinformatics data retrieval tasks. Searching rsIDs is not the only feature of high-perf-bio.  
  > **username_3**: Sure, your tool might be a jack of all trades but for the task at hand, is your tool faster - that is, for the same dataset, does MongoDB give you a significant advantage? In any case, my suggestion is based on the fact that you're not as well knows as username_0, so providing at least as much info as he does will only help your case.  
  
  ---
  
  ### Answer by username_4 on 2019-05-08. Score: 2
  
  I am not aware of any widespread index formats for query-by-QNAME for BAM or query-by-ID for VCF. I thought there were historical *samtools{-devel,-help}* threads to the effect of “in principle one could index a named-sorted BAM file, but there's insufficient demand so no-one's ever implemented it”, but I can't find any just now.
  
  Moreover note that VCF files are by definition sorted by genomic position, so such an index would just be a hash table of rsID to file offset or genomic position.
  
  If you are using dbSNP, probably you have a dbSNP database or API that you can use to query by rsID and get genomic position back. So you can retrieve your VCF contents by rsID in two stages: do that dbSNP query, and use its results to find the variants in your VCF file by position.
  
  
  ---
  
  ### Answer by username_5 on 2019-05-08. Score: 1
  
  Take a look at [`grabix`](https://github.com/arq5x/grabix). Using `grabix` you could implement a binary search on a VCF sorted by rsID and compressed using `bgzip`.
  
  
  ---
  
  