Answers.Body: <p>You don't have a RAM problem because you only have about:</p>
  
  <blockquote>
    <p>10-20 reads that I want to access from each file.</p>
  </blockquote>
  
  <p>Please go for (2). If you are concerned about RAM usage, you may selectively write some of your data back to disk.</p>
  
  <p>Tens of thousands of 10-20 reads is not a problem for modern computing. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-17T05:01:03.353Z
Answers.LastActivityDate: 2019-01-17T05:01:03.353Z
Answers.UserDisplayName: SmallChess
Body: <p>There was a nice question on <a href="https://bioinformatics.stackexchange.com/questions/807/">random access on a fastq file</a>, however I have the opposite problem. I want to programatically access non-random fastq entries quickly (using C++).</p>
  
  <p>Say I have several pairs of <code>fastq.gz</code> files that could be hundreds of millions or > 1 billion read pairs total. I know the number of read pairs in each file and I have a list of 10-20 reads that I want to access from each file. I only know the 0-based index of the fastq entry that I want to access, not the read ID. So if I want the  234th and 863rd reads the naive solution is to open the fastq file and scan through 864 records. I need to perform some tasks with these reads, then I move on and access/process another batch of reads.</p>
  
  <ul>
  <li>I need to perform the above task tens of thousands of times, so the options are:
  
  <ul>
  <li><ol>
  <li><strong>Low RAM usage. Many file IO ops. No disk usage:</strong> Opening/scanning/closing the <code>fastq.gz</code> files then processing the desired reads <strong>tens of thousands of times.</strong></li>
  </ol></li>
  <li><ol start="2">
  <li><strong>Extreme RAM usage. One round of file reading. No disk usage:</strong> Read in each <code>fastq.gz</code> file once, store all of the desired reads in RAM and access and process those reads as needed</li>
  </ol></li>
  <li><ol start="3">
  <li><strong>Low RAM usage. One round of file reading. Some disk usage:</strong> Coming up with an intermediate file format similar to a <code>bam</code> that will allow me to parse and sort the reads with their groups and other metadata later on.</li>
  </ol></li>
  </ul></li>
  </ul>
  
  <p>If it makes a difference the fastq.gz files are on spinning disks, not SSDs.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-16T23:52:04.890Z
Comments.Text: Load all lists into RAM and sequentially scan all FASTQ files.
Comments.UserDisplayName: user172818
Tags: <fastq><algorithms><c++>
Title: Non-random access on a fastq file
ThreadText: # Non-random access on a fastq file
  
  (Asked by: username_0 on 2019-01-16)
  
  There was a nice question on [random access on a fastq file](https://bioinformatics.stackexchange.com/questions/807/), however I have the opposite problem. I want to programatically access non-random fastq entries quickly (using C++).
  
  Say I have several pairs of `fastq.gz` files that could be hundreds of millions or > 1 billion read pairs total. I know the number of read pairs in each file and I have a list of 10-20 reads that I want to access from each file. I only know the 0-based index of the fastq entry that I want to access, not the read ID. So if I want the 234th and 863rd reads the naive solution is to open the fastq file and scan through 864 records. I need to perform some tasks with these reads, then I move on and access/process another batch of reads.
  
  * I need to perform the above task tens of thousands of times, so the options are:
    + 1. **Low RAM usage. Many file IO ops. No disk usage:** Opening/scanning/closing the `fastq.gz` files then processing the desired reads **tens of thousands of times.**
    + 2. **Extreme RAM usage. One round of file reading. No disk usage:** Read in each `fastq.gz` file once, store all of the desired reads in RAM and access and process those reads as needed
    + 3. **Low RAM usage. One round of file reading. Some disk usage:** Coming up with an intermediate file format similar to a `bam` that will allow me to parse and sort the reads with their groups and other metadata later on.
  
  If it makes a difference the fastq.gz files are on spinning disks, not SSDs.
  
  **Tags:** fastq, algorithms, c++
  
  > **Comments:**
  > **username_1**: Load all lists into RAM and sequentially scan all FASTQ files.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-01-17. Score: 2 | Accepted answer
  
  You don't have a RAM problem because you only have about:
  
  > 10-20 reads that I want to access from each file.
  
  Please go for (2). If you are concerned about RAM usage, you may selectively write some of your data back to disk.
  
  Tens of thousands of 10-20 reads is not a problem for modern computing.
  
  
  ---
  
  