Answers.Body: <p>I'd recommend having a go with either <a href="https://onecodex.com/" rel="nofollow noreferrer">OneCodex</a> (commercial / Freemium) or <a href="http://ccb.jhu.edu/software/centrifuge/" rel="nofollow noreferrer">Centrifuge</a>.</p>
  
  <p>Chris Mason's lab has <a href="https://genomebiology.biomedcentral.com/articles/10.1186/s13059-017-1299-7" rel="nofollow noreferrer">recently looked at metagenomic classification</a> and OneCodex gave the most accurate result.</p>
  
  <p>Centrifuge uses <a href="http://genome.cshlp.org/content/26/12/1721.full" rel="nofollow noreferrer">Burrows-Wheeler lookups</a>, but hasn't yet been compared by the Mason Lab.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-08T09:03:03.213Z
Answers.LastActivityDate: 2017-10-08T09:03:03.213Z
Answers.UserDisplayName: gringer
Body: <p>I'm a computer scientist working with biologists at a small school that doesn't have dedicated bioinformatics staff. I apologize if I use incorrect terminology since I have limited bioinformatics background.</p>
  
  <p>Our biology staff is studying a soil sample that has been sequenced. The resulting sequences (in the hundreds of millions) were analyzed using mg-rast, but they didn't seem to have a high confidence in the results.</p>
  
  <p>I've been asked to help analyze these sequences. I setup a local instance of NCBI blast, but it was far too slow to analyze that many sequences. We moved on to using BLAT. They've requested we use the nt database from NCBI. I converted the nt FASTA file to 2bit file, splitting the files into small chunks.</p>
  
  <p>gfServer seems to have limitations however. When I run gfServer, I get the following error while gfServer attempts to start.</p>
  
  <blockquote>
    <p>Exceeding 4 billion bases, sorry gfServer can't handle that.</p>
  </blockquote>
  
  <p>Is it possible to use the full nt database using BLAT? Is there an alternative to BLAT that can handle that many sequences?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-07T18:09:47.463Z
Comments.Text: What do they actually want to do with the sequences? See the closest match? All the matches? Something else entirely?
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-07T18:15:11.790Z
Comments.Text: See the closest match for each sequence, then generate a summary statistics about the sample's environment.
Comments.UserDisplayName: cbake
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-07T22:08:50.557Z
Comments.Text: The closest match can be done with STAR/hisat2/bowtie2/bwa or any other standard aligner that can handle large reference fasta files. That would prove vastly faster.
Comments.UserDisplayName: Devon Ryan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-08T20:19:47.057Z
Comments.Text: If you want to use BLAT, you could split by searches by chromosome.
Comments.UserDisplayName: Alex Reynolds
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-09T07:19:04.950Z
Comments.Text: Why use the entire `nt` database? If you're analyzing a soil sample, you already know you don't care about any mammalian sequences, for instance. You can make the search much faster if you choose a subset of nt instead of the entire thing. Especially since your biologists are probably only interested in unicellular species.
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-10-12T12:48:55.780Z
Comments.Text: If you want a drop in placement for blast that's just much faster, try DIAMOND
Comments.UserDisplayName: Chris_Rands
Tags: <phylogenetics><blast><metagenome><blat>
Title: Using large databases with BLAT
ThreadText: # Using large databases with BLAT
  
  (Asked by: username_0 on 2017-10-07)
  
  I'm a computer scientist working with biologists at a small school that doesn't have dedicated bioinformatics staff. I apologize if I use incorrect terminology since I have limited bioinformatics background.
  
  Our biology staff is studying a soil sample that has been sequenced. The resulting sequences (in the hundreds of millions) were analyzed using mg-rast, but they didn't seem to have a high confidence in the results.
  
  I've been asked to help analyze these sequences. I setup a local instance of NCBI blast, but it was far too slow to analyze that many sequences. We moved on to using BLAT. They've requested we use the nt database from NCBI. I converted the nt FASTA file to 2bit file, splitting the files into small chunks.
  
  gfServer seems to have limitations however. When I run gfServer, I get the following error while gfServer attempts to start.
  
  > Exceeding 4 billion bases, sorry gfServer can't handle that.
  
  Is it possible to use the full nt database using BLAT? Is there an alternative to BLAT that can handle that many sequences?
  
  **Tags:** phylogenetics, blast, metagenome, blat
  
  > **Comments:**
  > **username_1**: What do they actually want to do with the sequences? See the closest match? All the matches? Something else entirely?  
  > **username_0**: See the closest match for each sequence, then generate a summary statistics about the sample's environment.  
  > **username_1**: The closest match can be done with STAR/hisat2/bowtie2/bwa or any other standard aligner that can handle large reference fasta files. That would prove vastly faster.  
  > **username_2**: If you want to use BLAT, you could split by searches by chromosome.  
  > **username_3**: Why use the entire `nt` database? If you're analyzing a soil sample, you already know you don't care about any mammalian sequences, for instance. You can make the search much faster if you choose a subset of nt instead of the entire thing. Especially since your biologists are probably only interested in unicellular species.  
  > **username_4**: If you want a drop in placement for blast that's just much faster, try DIAMOND  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_5 on 2017-10-08. Score: 1
  
  I'd recommend having a go with either [OneCodex](https://onecodex.com/) (commercial / Freemium) or [Centrifuge](http://ccb.jhu.edu/software/centrifuge/).
  
  Chris Mason's lab has [recently looked at metagenomic classification](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-017-1299-7) and OneCodex gave the most accurate result.
  
  Centrifuge uses [Burrows-Wheeler lookups](http://genome.cshlp.org/content/26/12/1721.full), but hasn't yet been compared by the Mason Lab.
  
  
  ---
  
  