Answers.Body: <p>The question you linked is about storing genotypes. Your question is about keeping site information, which is different and much easier.</p>
  
  <p>As to solutions, you can parse the information you want and store them in a standard SQL database. You can use UCSC binning for region queries. Modern SQL databases all provide R-tree indices, which can be adapted for region query and is probably better these days.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-30T17:51:40.207Z
Answers.Text: A standard SQL database will be slow for such large amounts of data. I will be filtering on annotations as well as regions.
Answers.UserDisplayName: user2689782
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-30T18:15:20.950Z
Answers.Text: @user2689782 Have you tried? SQLite for example easily handles tens of millions of variants at my hand. UCSC/EnsEMBL keep the entire dbSNP in MySQL as well. The most efficient way to deal with annotations highly depends on the type of annotations and queries. It is more about schema design.
Answers.UserDisplayName: user172818
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-30T17:12:39.923Z
Answers.LastActivityDate: 2019-05-30T17:12:39.923Z
Answers.UserDisplayName: user172818
Body: <p>I need to store a large number of variants with their annotations. I need to import dbSNP, as well as Clinvar, and ExAC etc. Also I have sample variant data from patients. I need to be able to search on genomic regions as well as annotations. And it needs to be scalable.
  What would be a preferred data storage technology for such a case? What is the industry using? It needs to be accessible by APIs/REST.
  We are using ElasticSearch, but that is proving to be expensive in terms of servers needed.</p>
  
  <p>Something similar to this has been discussed in 
  <a href="https://bioinformatics.stackexchange.com/questions/195/the-state-limitations-and-comparisons-of-large-variant-stores">The state, limitations and comparisons of large variant stores</a>, but that post is a little old, I am sure there has been progress since then.</p>
  
Tags: <database><snp><indel>
Title: Store and search on variant and annotation data
ThreadText: # Store and search on variant and annotation data
  
  (Asked by: username_0 on 2019-05-30)
  
  I need to store a large number of variants with their annotations. I need to import dbSNP, as well as Clinvar, and ExAC etc. Also I have sample variant data from patients. I need to be able to search on genomic regions as well as annotations. And it needs to be scalable.
  What would be a preferred data storage technology for such a case? What is the industry using? It needs to be accessible by APIs/REST.
  We are using ElasticSearch, but that is proving to be expensive in terms of servers needed.
  
  Something similar to this has been discussed in
  [The state, limitations and comparisons of large variant stores](https://bioinformatics.stackexchange.com/questions/195/the-state-limitations-and-comparisons-of-large-variant-stores), but that post is a little old, I am sure there has been progress since then.
  
  **Tags:** database, snp, indel
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-30. Score: 1
  
  The question you linked is about storing genotypes. Your question is about keeping site information, which is different and much easier.
  
  As to solutions, you can parse the information you want and store them in a standard SQL database. You can use UCSC binning for region queries. Modern SQL databases all provide R-tree indices, which can be adapted for region query and is probably better these days.
  
  > **Comments:**
  > **username_0**: A standard SQL database will be slow for such large amounts of data. I will be filtering on annotations as well as regions.  
  > **username_1**: @username_0 Have you tried? SQLite for example easily handles tens of millions of variants at my hand. UCSC/EnsEMBL keep the entire dbSNP in MySQL as well. The most efficient way to deal with annotations highly depends on the type of annotations and queries. It is more about schema design.  
  
  ---
  
  