Answers.Body: <p>It took me a while to figure out that the "<strong>index</strong>" is the same thing as the "<strong>barcode</strong>" that says which sample each sequence is from on a multiplexed run.  </p>
  
  <p>If your data is <em>not demultiplexed</em> (single <code>R1.fastq</code> and <code>R2.fastq</code> files contain the information for multiple samples), then this <code>I1.fastq</code> file is what you use to assign each sequence to a sample (ie to "demultiplex"). The spreadsheet listing which barcode/index is which sample is often called a "mapping file."</p>
  
  <p>If your data is <em>already demultiplexed</em> (you have separate <code>R1.fastq</code> and <code>R2.fastq</code> files for each sample), then you don't really need it... however some people use the index sequences in their quality control workflows. For example, this 2016 paper in BMC Genomics by Wright and Vetsigien "Quality filtering of Illumina index reads mitigates sample cross-talk": <a href="https://doi.org/10.1186/s12864-016-3217-x" rel="nofollow noreferrer">https://doi.org/10.1186/s12864-016-3217-x</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T14:19:43.590Z
Answers.Text: The term index is almost always used to refer to the 8-bp sequence that the Illumina machine reads to identify the library-of-origin of the individual read, while the term barcode could also mean some sequence within R1 or R2 itself that the user added to label provenance - such as being from a specific cell or specific amplicon.
Answers.UserDisplayName: conchoecia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T17:31:39.927Z
Answers.Text: Yes, in the context of a 10xGenomics single cell run, the "barcode" means the cell barcode, which is not at all the same thing as the sample index.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T19:17:27.160Z
Answers.Text: I received 3 files that exactly match the file names listed by the original poster, except that my data was clearly not demultiplexed yet. My index file referred to what my mapping file listed as the "SampleBarcode".  *It's certainly possible that formats are different for single-cell data.* For perspective, my data is from a multiplexed 16S rRNA gene amplicon miSeq run. My barcodes identified which sample each sequence was from, and they were a 12 bp sequence attached to the forward primer (so it was not dual-indexed). This is the protocol used by the Earth Microbiome Project.
Answers.UserDisplayName: rrr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T00:06:29.563Z
Answers.LastActivityDate: 2019-02-12T00:06:29.563Z
Answers.UserDisplayName: rrr
Answers.Body: <p>I looked at 10x data a couple of years ago, and at that time, their protocol would use four separate indexes, with balanced index sequences, as a way of introducing diversity into the library.  These four indexes were separate from the individual oligos used to mark single cells.  For an individual sample, you would need to combine the reads from all four separate indexes, which is how it looks like in your data.  There is a brief explanation here: <a href="https://support.10xgenomics.com/single-cell-gene-expression/index/doc/specifications-sample-index-sets-for-single-cell-3" rel="nofollow noreferrer">https://support.10xgenomics.com/single-cell-gene-expression/index/doc/specifications-sample-index-sets-for-single-cell-3</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T10:13:59.027Z
Answers.LastActivityDate: 2019-02-12T10:13:59.027Z
Answers.UserDisplayName: Jay Moore
Answers.Body: <h1>tldr - The I*.fastq.gz file contains the read index sequences.</h1>
  <h2>long explanation</h2>
  <p>Illumina uses a program called <a href="https://support.illumina.com/content/dam/illumina-support/documents/documentation/software_documentation/bcl2fastq/bcl2fastq_letterbooklet_15038058brpmi.pdf" rel="noreferrer"><code>bcl2fastq</code></a> to demultiplex sequencing runs.</p>
  <p>This software takes a list of samples and their associated indices and uses those sequences to make one or more fastq files per sample, binned by one or two index sequences on either end of the sequencing molecule (<a href="https://support.illumina.com/content/dam/illumina-support/documents/documentation/system_documentation/miseq/indexed-sequencing-overview-guide-15057455-04.pdf" rel="noreferrer">i5 and i7 indices, see page 6 for HiSeq</a>).</p>
  <p>Illumina sequencing isn't perfect though, and there are sometimes errors in reading the index sequence. For example this index, <code>CAGCCCAC</code> can easily have read errors in the <code>A</code> sandwiched between many <code>C</code>s: <code>CAGCCCAC -&gt; CAGCCCCC</code>.</p>
  <p>Instead of throwing out all of the reads with indexing sequencing errors, the <code>bcl2fastq</code> program includes reads that are discernibly derived from a true index for that sample as long as there is no overlap with another sample.</p>
  <p>In the example provided above, the four indices for <code>mysample</code> were: <code>ACATTACT</code>, <code>CAGCCCAC</code>, <code>GGCAATGG</code>, <code>TTTGGGTA</code>. If we look at all of the index sequences in the <code>Mysample_I1_001.fastq.gz</code> file, we will see that sequences with the four correct indices are indeed the most prevalent, however there are also reads that have indices with sequencing errors derived from the correct indices.</p>
  <pre><code>bioawk -cfastx '{print($seq)}' Mysample_I1_001.fastq.gz | \
      sort | uniq -c | sort -k1 -nr | head
  
  count      index       source
  41362311   CAGCCCAC    True index
  37209190   GGCAATGG    True index
  36863213   ACATTACT    True index 
  33674467   TTTGGGTA    True index
   1140358   NAGCCCAC    CAGCCCAC
   1026099   NGCAATGG    GGCAATGG
   1016754   NCATTACT    ACATTACT
    933342   NTTGGGTA    TTTGGGTA
    119626   TTTGGGGA    TTTGGGTA
     98657   GTTGGGTA    TTTGGGTA
     96625   GGCAATGA    GGCAATGG
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-08T19:20:25.627Z
Answers.LastActivityDate: 2018-10-08T19:20:25.627Z
Answers.LastEditDate: 2020-06-18T08:30:41.743Z
Answers.UserDisplayName: conchoecia
Body: <p>What is the index fastq file that comes with some Illumina sequencing datasets? (The <code>samplename_I*.fastq.gz</code> file.)</p>
  
  <p>For example, I recently received some 10X Chromium reads for two libraries sequenced on the same lane. This was a 2x150 sequencing run, so there should be two fastq files. The sequencing center demultiplexed the libraries and generated two separate directories - one for each library.</p>
  
  <ul>
  <li>In each directory there are three fastq files:
  
  <ul>
  <li><code>Mysample_I1_001.fastq.gz</code></li>
  <li><code>Mysample_R1_001.fastq.gz</code></li>
  <li><code>Mysample_R2_001.fastq.gz</code></li>
  </ul></li>
  </ul>
  
  <p>I know that the <a href="https://support.illumina.com/content/dam/illumina-support/documents/documentation/system_documentation/miseq/indexed-sequencing-overview-guide-15057455-04.pdf" rel="noreferrer">R1 fastq file contains the forward reads and the R2 fastq file contains the reverse reads</a>, but what is the index file? The first few sequences do not match the 10X indices for this library (<code>ACATTACT</code>, <code>CAGCCCAC</code>, <code>GGCAATGG</code>, <code>TTTGGGTA</code>).</p>
  
  <pre><code>&gt;bioawk -cfastx '{print($seq)}' Mysample_I1_001.fastq.gz | head
  NTTGGGTA
  NGCAATGG
  NAGCCCAC
  NAGCCCAC
  NCATTACT
  NCATTACT
  NGCAATGG
  NAGCCCAC
  NTTGGGTA
  NAGCCCAC
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-08T23:30:03.100Z
Comments.Text: I don't know 10x Chromium software, but I know their single cell pipeline, and many versions ago, the software was expecting the read indices to be in a separate file along with the reads themselves.  Cellranger has since changed their pipeline so that it doesn't want these anymore, but whoever gave you your files might have seen that it was a 10xGenomics protocol, and decided to give you the indices like that just in case you needed them,
Comments.UserDisplayName: swbarnes2
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-09T08:19:41.110Z
Comments.Text: For what it's worth, 10X doesn't require the index files as input (you can just give it demultiplexed samples) and Illumina's software won't produce those index files by default. Also, you can have up to 2 index files for Illumina data (though you typically won't if you're making standard 10X libraries).
Comments.UserDisplayName: Devon Ryan
Tags: <ngs><illumina><10x-genomics><demultiplexing>
Title: What is the index fastq file (sample_I*.fastq.gz) generated when demultiplexing Illumina paired-end runs?
ThreadText: # What is the index fastq file (sample_I*.fastq.gz) generated when demultiplexing Illumina paired-end runs?
  
  (Asked by: username_0 on 2018-10-08)
  
  What is the index fastq file that comes with some Illumina sequencing datasets? (The `samplename_I*.fastq.gz` file.)
  
  For example, I recently received some 10X Chromium reads for two libraries sequenced on the same lane. This was a 2x150 sequencing run, so there should be two fastq files. The sequencing center demultiplexed the libraries and generated two separate directories - one for each library.
  
  * In each directory there are three fastq files:
    + `Mysample_I1_001.fastq.gz`
    + `Mysample_R1_001.fastq.gz`
    + `Mysample_R2_001.fastq.gz`
  
  I know that the [R1 fastq file contains the forward reads and the R2 fastq file contains the reverse reads](https://support.illumina.com/content/dam/illumina-support/documents/documentation/system_documentation/miseq/indexed-sequencing-overview-guide-15057455-04.pdf), but what is the index file? The first few sequences do not match the 10X indices for this library (`ACATTACT`, `CAGCCCAC`, `GGCAATGG`, `TTTGGGTA`).
  
  ```
  >bioawk -cfastx '{print($seq)}' Mysample_I1_001.fastq.gz | head
  NTTGGGTA
  NGCAATGG
  NAGCCCAC
  NAGCCCAC
  NCATTACT
  NCATTACT
  NGCAATGG
  NAGCCCAC
  NTTGGGTA
  NAGCCCAC
  
  ```
  
  **Tags:** ngs, illumina, 10x-genomics, demultiplexing
  
  > **Comments:**
  > **username_1**: I don't know 10x Chromium software, but I know their single cell pipeline, and many versions ago, the software was expecting the read indices to be in a separate file along with the reads themselves.  Cellranger has since changed their pipeline so that it doesn't want these anymore, but whoever gave you your files might have seen that it was a 10xGenomics protocol, and decided to give you the indices like that just in case you needed them,  
  > **username_2**: For what it's worth, 10X doesn't require the index files as input (you can just give it demultiplexed samples) and Illumina's software won't produce those index files by default. Also, you can have up to 2 index files for Illumina data (though you typically won't if you're making standard 10X libraries).  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2018-10-08. Score: 10 | Accepted answer
  
  # tldr - The I\*.fastq.gz file contains the read index sequences.
  
  ## long explanation
  
  Illumina uses a program called [`bcl2fastq`](https://support.illumina.com/content/dam/illumina-support/documents/documentation/software_documentation/bcl2fastq/bcl2fastq_letterbooklet_15038058brpmi.pdf) to demultiplex sequencing runs.
  
  This software takes a list of samples and their associated indices and uses those sequences to make one or more fastq files per sample, binned by one or two index sequences on either end of the sequencing molecule ([i5 and i7 indices, see page 6 for HiSeq](https://support.illumina.com/content/dam/illumina-support/documents/documentation/system_documentation/miseq/indexed-sequencing-overview-guide-15057455-04.pdf)).
  
  Illumina sequencing isn't perfect though, and there are sometimes errors in reading the index sequence. For example this index, `CAGCCCAC` can easily have read errors in the `A` sandwiched between many `C`s: `CAGCCCAC -> CAGCCCCC`.
  
  Instead of throwing out all of the reads with indexing sequencing errors, the `bcl2fastq` program includes reads that are discernibly derived from a true index for that sample as long as there is no overlap with another sample.
  
  In the example provided above, the four indices for `mysample` were: `ACATTACT`, `CAGCCCAC`, `GGCAATGG`, `TTTGGGTA`. If we look at all of the index sequences in the `Mysample_I1_001.fastq.gz` file, we will see that sequences with the four correct indices are indeed the most prevalent, however there are also reads that have indices with sequencing errors derived from the correct indices.
  
  ```
  bioawk -cfastx '{print($seq)}' Mysample_I1_001.fastq.gz | \
      sort | uniq -c | sort -k1 -nr | head
  
  count      index       source
  41362311   CAGCCCAC    True index
  37209190   GGCAATGG    True index
  36863213   ACATTACT    True index 
  33674467   TTTGGGTA    True index
   1140358   NAGCCCAC    CAGCCCAC
   1026099   NGCAATGG    GGCAATGG
   1016754   NCATTACT    ACATTACT
    933342   NTTGGGTA    TTTGGGTA
    119626   TTTGGGGA    TTTGGGTA
     98657   GTTGGGTA    TTTGGGTA
     96625   GGCAATGA    GGCAATGG
  
  ```
  
  
  ---
  
  ### Answer by username_3 on 2019-02-12. Score: 2
  
  It took me a while to figure out that the "**index**" is the same thing as the "**barcode**" that says which sample each sequence is from on a multiplexed run.
  
  If your data is *not demultiplexed* (single `R1.fastq` and `R2.fastq` files contain the information for multiple samples), then this `I1.fastq` file is what you use to assign each sequence to a sample (ie to "demultiplex"). The spreadsheet listing which barcode/index is which sample is often called a "mapping file."
  
  If your data is *already demultiplexed* (you have separate `R1.fastq` and `R2.fastq` files for each sample), then you don't really need it... however some people use the index sequences in their quality control workflows. For example, this 2016 paper in BMC Genomics by Wright and Vetsigien "Quality filtering of Illumina index reads mitigates sample cross-talk": <https://doi.org/10.1186/s12864-016-3217-x>
  
  > **Comments:**
  > **username_0**: The term index is almost always used to refer to the 8-bp sequence that the Illumina machine reads to identify the library-of-origin of the individual read, while the term barcode could also mean some sequence within R1 or R2 itself that the user added to label provenance - such as being from a specific cell or specific amplicon.  
  > **username_1**: Yes, in the context of a 10xGenomics single cell run, the "barcode" means the cell barcode, which is not at all the same thing as the sample index.  
  > **username_3**: I received 3 files that exactly match the file names listed by the original poster, except that my data was clearly not demultiplexed yet. My index file referred to what my mapping file listed as the "SampleBarcode".  *It's certainly possible that formats are different for single-cell data.* For perspective, my data is from a multiplexed 16S rRNA gene amplicon miSeq run. My barcodes identified which sample each sequence was from, and they were a 12 bp sequence attached to the forward primer (so it was not dual-indexed). This is the protocol used by the Earth Microbiome Project.  
  
  ---
  
  ### Answer by username_4 on 2019-02-12. Score: 0
  
  I looked at 10x data a couple of years ago, and at that time, their protocol would use four separate indexes, with balanced index sequences, as a way of introducing diversity into the library. These four indexes were separate from the individual oligos used to mark single cells. For an individual sample, you would need to combine the reads from all four separate indexes, which is how it looks like in your data. There is a brief explanation here: <https://support.10xgenomics.com/single-cell-gene-expression/index/doc/specifications-sample-index-sets-for-single-cell-3>
  
  
  ---
  
  