Answers.Body: <p>I'm late to the party but simply run <code>cellranger mkfastq</code> twice, with different arguments for the mask and with <code>--filter-dual-index</code> for the double-indexed samples. In a second step you then have to disentangle the dual A+B reads from single-index A reads based on the readID's that are found in both of the demultiplexings. This is cumbersome and time+diskspace consuming, but it can be done. I would use some Unix tools for that but there are probably smarter ways.</p>
  <p>EDIT: I mistakenly said <code>bcl2fastq</code>, this must be <code>cellranger mkfastq</code>. (thanks AmadeusDrZaius)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-25T18:46:23.320Z
Answers.Text: Thanks for your suggestion. I only see that "--filter-dual-index" option available for `cellranger mkfastq`, rather than bcl2fastq itself. Is that where you saw the option?
Answers.UserDisplayName: AmadeusDrZaius
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-26T19:54:49.303Z
Answers.Text: ah yes, that is true, sorry. I guess you would have to do an additional filtering step on the FASTQ file, the second word on the ID lines is 'index1+index2' for properly demultiplexed reads.
Answers.UserDisplayName: plijnzaad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-21T08:57:08.707Z
Answers.LastActivityDate: 2021-05-26T19:58:00.250Z
Answers.LastEditDate: 2021-05-26T19:58:00.250Z
Answers.UserDisplayName: plijnzaad
Answers.Body: <p>Have you just tried giving bcl2fastq one sample sheet with a mix of single and dual indices?  I don't think what you are trying to do is a problem.</p>
  
  <blockquote>
    <p>if any single index A is part of a dual index A+B,</p>
  </blockquote>
  
  <p>Well, it might be a problem if you did that.  That was poor planning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T18:01:30.210Z
Answers.Text: That was the first thing tried but unfortunately bcl2fastq will give the error "Barcodes have an unequal number of components" when you try to do that (using v2.17). I'll add a note to my question.
Answers.UserDisplayName: AmadeusDrZaius
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T19:19:04.603Z
Answers.Text: Are you sure that the single index samples don't have a site for the second index to sit down and read?  My lab does some samples with single indicies and some with double, but they always prep with libraries with kits meant for dual indexing, so the second index is always there, even if I ignore it.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T19:33:06.290Z
Answers.Text: That's a good idea that might be a good workaround in some cases, but I think the general answer is that they are not always physically dual-indexed. The use of "N" as described in the bcl2fastq documentation would work if not for the fact that a bug in bcl2fastq causes it to treat "N" as a literal base "N" instead of as a wildcard. Given that, I'm guessing that the Picard tools are more likely to have a full solution for this particular situation.
Answers.UserDisplayName: AmadeusDrZaius
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T19:51:44.690Z
Answers.Text: I don't think it's going to be trivial for software to figure out the difference between "second index doesn't match what I expect" and "second index isn't really here".  You might have to have bcl2fastq export the indices as fastqs, so you can use the quality score to parse yourself.
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T20:10:26.517Z
Answers.Text: That's an interesting point. I expected that it would just make a "best guess", but I can see why it wouldn't. It sounds like that would primarily only be an issue in the case of "A" and "A+B" that I described above. If the case of "A" and "A+B" is avoided, it should be able to tell the difference. Thanks for your thoughts.
Answers.UserDisplayName: AmadeusDrZaius
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T20:37:40.770Z
Answers.Text: I would be surprised if the Experiment Manager software would allow a run configuration that was un-processable.  I take it the instrument was run without a sample sheet made by Experiment Manager?
Answers.UserDisplayName: swbarnes2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-30T16:15:38.150Z
Answers.LastActivityDate: 2019-07-30T16:15:38.150Z
Answers.UserDisplayName: swbarnes2
Body: <h2>The problem</h2>
  
  <p>If I have a sample sheet that contains both single-indexed and dual-indexed samples, I can split it up into two sample sheets and then run bcl2fastq on each one. However, when doing this, large Undetermined fastq files are generated. E.g., when processing the single-indexed samples, all the dual-indexed samples go to Undetermined. And when processing the dual-indexed samples, the single-indexed samples go to Undetermined.</p>
  
  <p>Additionally, because they are being processed separately, if any single index A is part of a dual index A+B, then when processing the single indexes, an A+B may be mistaken for an A, so it would seem that they need to processed simultaneously to avoid this mis-assignment.</p>
  
  <h2>The question</h2>
  
  <p>Given a sample sheet and directories of BCL files, how can such a set of sequencing data be demultiplexed correctly, either using bcl2fastq or the Picard tools?</p>
  
  <p>To put it another way, I want to demultiplex a single sequencing run that contains both single-indexed and dual-indexed samples. It can be assumed that the indexes are sufficiently distinct such that any sample's index configuration is different from any other. But assuming that the different index configurations are not segregated to particular lanes of the sequencer, the question is how to demultiplex the files correctly such that both the single-indexed and dual-indexed samples are recognized.</p>
  
  <h2>Attempts at a solution</h2>
  
  <h3>Using bcl2fastq directly</h3>
  
  <p>If a sample sheet of a mix of indexes is given to bcl2fastq (v2.17) directly, it produces the error</p>
  
  <pre><code>ERROR: bcl2fastq::common::Exception: Success (0): .../bcl2fastq2/src/cxx/lib/layout/BarcodeCollisionDetector.cpp(127): 
  Throw in function void bcl2fastq::layout::BarcodeCollisionDetector::validateNewBarcodeSizesAgainstExisting(const std::vector&lt;long unsigned int&gt;&amp;) const
  Dynamic exception type: boost::exception_detail::clone_impl&lt;bcl2fastq::layout::BarcodeCollisionError&gt;
  std::exception::what: Barcodes have an unequal number of components.
  </code></pre>
  
  <blockquote>
    <p>Barcodes have an unequal number of components.</p>
  </blockquote>
  
  <h3>Using picard</h3>
  
  <p>It seems that it should be possible using the Picard tools, but I have not found a way to set up the inputs to ExtractIlluminaBarcodes and IlluminaBasecallsToFastq that processes this configuration correctly.</p>
  
  <p>The syntax for supporting multiple index configurations is not entirely clear. But when using various combinations of N and '*' on multiplex_params.tsv and barcodes.txt required by the Picard tools, a large Undetermined file is still produced and actual sample fastq files are tiny, indicating it is not processing them correctly.</p>
  
  <h3>Using multiple calls to bcl2fastq</h3>
  
  <p>As indicated in the discussion above, this suffers from the problem of having all the dual indexed samples going to the "Undetermined" file when processing the single-indexed samples, or vice versa, and creates two output directories of reports and stats, which must be merged.</p>
  
  <h3>Padding empty indexes with N</h3>
  
  <p>By padding, I mean converting ACGTACGT to ACGTACGT+NNNNNNNN so that the single-index samples in the sample sheet are "dual" as well. This strategy seems that it would work except there is a bug in bcl2fastq that it treats "N" literally instead of as a wildcard. <a href="https://support.illumina.com/content/dam/illumina-support/documents/downloads/software/bcl2fastq/bcl2fastq2-2-20-0-release-notes-1000000039216-00.pdf" rel="nofollow noreferrer">See the release notes for details.</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-30T15:27:48.643Z
Comments.Text: Welcome to the site. Sorry but I don't understand which files do you have. Could you clarify what are your files? Or do you have a mix of single-indexed and dual-indexed and you can't differentiate between them?
Comments.UserDisplayName: llrs
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-30T15:33:11.050Z
Comments.Text: @llrs Hi, thanks for the question. In this case, I'm trying to demultiplex a single sequencing run that contains both single-indexed and dual-indexed samples. You can assume that the indexes are sufficiently distinct such that any sample's index configuration is different from any other. But assuming that the different index configurations are not segregated to particular lanes of the sequencer, the question is how to demultiplex the files correctly such that both the single-indexed and dual-indexed samples are recognized.
Comments.UserDisplayName: AmadeusDrZaius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-30T15:34:32.993Z
Comments.Text: Ok, now I understood the question, but I'm sorry I can't help with it. Perhaps you could [edit] your last comment in the question for other users.
Comments.UserDisplayName: llrs
Tags: <illumina><picard><demultiplexing><indexing>
Title: How to demultiplex a mix of single-indexed and dual-indexed samples
ThreadText: # How to demultiplex a mix of single-indexed and dual-indexed samples
  
  (Asked by: username_0 on 2019-07-30)
  
  ## The problem
  
  If I have a sample sheet that contains both single-indexed and dual-indexed samples, I can split it up into two sample sheets and then run bcl2fastq on each one. However, when doing this, large Undetermined fastq files are generated. E.g., when processing the single-indexed samples, all the dual-indexed samples go to Undetermined. And when processing the dual-indexed samples, the single-indexed samples go to Undetermined.
  
  Additionally, because they are being processed separately, if any single index A is part of a dual index A+B, then when processing the single indexes, an A+B may be mistaken for an A, so it would seem that they need to processed simultaneously to avoid this mis-assignment.
  
  ## The question
  
  Given a sample sheet and directories of BCL files, how can such a set of sequencing data be demultiplexed correctly, either using bcl2fastq or the Picard tools?
  
  To put it another way, I want to demultiplex a single sequencing run that contains both single-indexed and dual-indexed samples. It can be assumed that the indexes are sufficiently distinct such that any sample's index configuration is different from any other. But assuming that the different index configurations are not segregated to particular lanes of the sequencer, the question is how to demultiplex the files correctly such that both the single-indexed and dual-indexed samples are recognized.
  
  ## Attempts at a solution
  
  ### Using bcl2fastq directly
  
  If a sample sheet of a mix of indexes is given to bcl2fastq (v2.17) directly, it produces the error
  
  ```
  ERROR: bcl2fastq::common::Exception: Success (0): .../bcl2fastq2/src/cxx/lib/layout/BarcodeCollisionDetector.cpp(127): 
  Throw in function void bcl2fastq::layout::BarcodeCollisionDetector::validateNewBarcodeSizesAgainstExisting(const std::vector<long unsigned int>&) const
  Dynamic exception type: boost::exception_detail::clone_impl<bcl2fastq::layout::BarcodeCollisionError>
  std::exception::what: Barcodes have an unequal number of components.
  
  ```
  
  > Barcodes have an unequal number of components.
  
  ### Using picard
  
  It seems that it should be possible using the Picard tools, but I have not found a way to set up the inputs to ExtractIlluminaBarcodes and IlluminaBasecallsToFastq that processes this configuration correctly.
  
  The syntax for supporting multiple index configurations is not entirely clear. But when using various combinations of N and '\*' on multiplex\_params.tsv and barcodes.txt required by the Picard tools, a large Undetermined file is still produced and actual sample fastq files are tiny, indicating it is not processing them correctly.
  
  ### Using multiple calls to bcl2fastq
  
  As indicated in the discussion above, this suffers from the problem of having all the dual indexed samples going to the "Undetermined" file when processing the single-indexed samples, or vice versa, and creates two output directories of reports and stats, which must be merged.
  
  ### Padding empty indexes with N
  
  By padding, I mean converting ACGTACGT to ACGTACGT+NNNNNNNN so that the single-index samples in the sample sheet are "dual" as well. This strategy seems that it would work except there is a bug in bcl2fastq that it treats "N" literally instead of as a wildcard. [See the release notes for details.](https://support.illumina.com/content/dam/illumina-support/documents/downloads/software/bcl2fastq/bcl2fastq2-2-20-0-release-notes-1000000039216-00.pdf)
  
  **Tags:** illumina, picard, demultiplexing, indexing
  
  > **Comments:**
  > **username_1**: Welcome to the site. Sorry but I don't understand which files do you have. Could you clarify what are your files? Or do you have a mix of single-indexed and dual-indexed and you can't differentiate between them?  
  > **username_0**: @username_1 Hi, thanks for the question. In this case, I'm trying to demultiplex a single sequencing run that contains both single-indexed and dual-indexed samples. You can assume that the indexes are sufficiently distinct such that any sample's index configuration is different from any other. But assuming that the different index configurations are not segregated to particular lanes of the sequencer, the question is how to demultiplex the files correctly such that both the single-indexed and dual-indexed samples are recognized.  
  > **username_1**: Ok, now I understood the question, but I'm sorry I can't help with it. Perhaps you could [edit] your last comment in the question for other users.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-05-21. Score: 1
  
  I'm late to the party but simply run `cellranger mkfastq` twice, with different arguments for the mask and with `--filter-dual-index` for the double-indexed samples. In a second step you then have to disentangle the dual A+B reads from single-index A reads based on the readID's that are found in both of the demultiplexings. This is cumbersome and time+diskspace consuming, but it can be done. I would use some Unix tools for that but there are probably smarter ways.
  
  EDIT: I mistakenly said `bcl2fastq`, this must be `cellranger mkfastq`. (thanks username_0)
  
  > **Comments:**
  > **username_0**: Thanks for your suggestion. I only see that "--filter-dual-index" option available for `cellranger mkfastq`, rather than bcl2fastq itself. Is that where you saw the option?  
  > **username_2**: ah yes, that is true, sorry. I guess you would have to do an additional filtering step on the FASTQ file, the second word on the ID lines is 'index1+index2' for properly demultiplexed reads.  
  
  ---
  
  ### Answer by username_3 on 2019-07-30. Score: 0
  
  Have you just tried giving bcl2fastq one sample sheet with a mix of single and dual indices? I don't think what you are trying to do is a problem.
  
  > if any single index A is part of a dual index A+B,
  
  Well, it might be a problem if you did that. That was poor planning.
  
  > **Comments:**
  > **username_0**: That was the first thing tried but unfortunately bcl2fastq will give the error "Barcodes have an unequal number of components" when you try to do that (using v2.17). I'll add a note to my question.  
  > **username_3**: Are you sure that the single index samples don't have a site for the second index to sit down and read?  My lab does some samples with single indicies and some with double, but they always prep with libraries with kits meant for dual indexing, so the second index is always there, even if I ignore it.  
  > **username_0**: That's a good idea that might be a good workaround in some cases, but I think the general answer is that they are not always physically dual-indexed. The use of "N" as described in the bcl2fastq documentation would work if not for the fact that a bug in bcl2fastq causes it to treat "N" as a literal base "N" instead of as a wildcard. Given that, I'm guessing that the Picard tools are more likely to have a full solution for this particular situation.  
  > **username_3**: I don't think it's going to be trivial for software to figure out the difference between "second index doesn't match what I expect" and "second index isn't really here".  You might have to have bcl2fastq export the indices as fastqs, so you can use the quality score to parse yourself.  
  > **username_0**: That's an interesting point. I expected that it would just make a "best guess", but I can see why it wouldn't. It sounds like that would primarily only be an issue in the case of "A" and "A+B" that I described above. If the case of "A" and "A+B" is avoided, it should be able to tell the difference. Thanks for your thoughts.  
  > **username_3**: I would be surprised if the Experiment Manager software would allow a run configuration that was un-processable.  I take it the instrument was run without a sample sheet made by Experiment Manager?  
  
  ---
  
  