Answers.Body: <p>EDIT: <strong><a href="https://bioinformatics.stackexchange.com/a/11311/4980">Chris_Rands's answer</a> is more authoritative and clarifies that this behavior is essentially a bug.</strong>  (I had only tested my example on Python 3.6.8 and hadn't seen PEP 479.)</p>
  
  <p>When <code>next</code> runs out of lines, it raises <code>StopIteration</code>, and apparently that stops the while loop similarly to calling <code>break</code> without you needing to catch the exception.  I couldn't see a definitive source on that but you can see it in action if you catch and re-raise the exception:</p>
  
  <pre><code>def ingest2(stream1, stream2):
      while True:
          try:
              record1, record2 = list(), list()
              for _ in range(4):
                  record1.append(next(stream1))
                  record2.append(next(stream2))
              yield record1, record2
          except StopIteration:
              print("StopIteration raised")
              raise
  </code></pre>
  
  <p>You can also try to catch it outside the loop, but it doesn't get re-raised at that point.  (If you try raising one yourself in a while loop, though, it <em>doesn't</em> catch it in the same way, so there must be some extra Python magic going on.  I can't see any difference between the two exception objects so I'd be curious what's going on there.)  Neat example.  You could also make a pair of FASTQ parsers using Biopython's <a href="https://biopython.org/wiki/SeqIO#sequence-input" rel="nofollow noreferrer">SeqIO.parse</a> for more complex/permanent situations if needed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-04T21:42:24.807Z
Answers.LastActivityDate: 2020-02-05T15:46:30.730Z
Answers.LastEditDate: 2020-02-05T15:46:30.730Z
Answers.UserDisplayName: Jesse
Answers.Body: <p>Your code only runs prior to Python 3.7, now it raises <code>RuntimeError: generator raised StopIteration</code>. This is essentially a bug fix described in <a href="https://www.python.org/dev/peps/pep-0479/" rel="nofollow noreferrer">PEP 479 -- Change StopIteration handling inside generators</a>. You can enable the new behaviour from Python 3.5 via <code>from __future__ import generator_stop</code>.</p>
  
  <p>Note, it's not related to file-handles per se, you can reproduce it with a smaller example with a simple iterator:</p>
  
  <pre><code>it = iter([1,2])
  
  def f(it):
      while True:
          yield next(it)
  
  print(list(f(it)))
  </code></pre>
  
  <p>You should not write code relying on the bug</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-04T21:54:48.543Z
Answers.LastActivityDate: 2020-02-04T21:54:48.543Z
Answers.UserDisplayName: Chris_Rands
Body: <p>I wrote some throwaway code for ingesting paired-end reads from a Fastq file into memory.</p>
  
  <pre class="lang-py prettyprint-override"><code>def ingest(stream1, stream2):
      while True:
          record1, record2 = list(), list()
          for _ in range(4):
              record1.append(next(stream1))
              record2.append(next(stream2))
          yield record1, record2
  </code></pre>
  
  <p>I had initially put the <code>for</code> loop in a <code>try/Except</code> block, anticipating I'd need to explicitly break out of the endless <code>while</code> loop. However, the code seems to work correctly even after I removed it. I even "corrupted" the final read in one of the paired files, and the <code>ingest</code> function performs reasonably (the last read pair is never <code>yield</code>ed).</p>
  
  <p>My question is...<strong>why does this code work</strong>, since I never explicitly break out of the loop?</p>
  
  <p>Here's an example of how the code is invoked.</p>
  
  <pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; with open('seqR1.fastq', 'r') as fh1, open('seqR2.fastq', 'r') as fh2:
  ...     counter = 0
  ...     for r1, r2 in interleave(fh1, fh2):
  ...         counter += 1
  &gt;&gt;&gt; counter
  5000
  &gt;&gt;&gt; r1
  ['@read5000/1\n', 'AGGTCTGGGGACAGGTGTTTTACAGGCTCCTTTAGGGGTGGCAAGTCTCCCGGAAACATGATTAGTGAAGACACGTTAACAGCGAAACTGACAGAGAACCTCAACATTCTCTGGTAGGAAAAGCCAGGATCCTGGCAGAGGAAGCAACGATTCGAGCAGGGAACTGTGCAGGACAGAGAACAATGCGGCCCCAATTTTGATCCAGGGGTAGTTAGCTTAGCTAAGGAGTTCCCCAGTCCCTAAGTGGAGAGGGGATCCAGAACCCTACAATTTATTATTATATCTTAGCGGATATATCTAT\n', '+\n', 'CCCCCGGGEGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@GGGGGGGFAGGGCGGGGGFGGGGGGGGGCGGGGGDGGGGFGGGGGGGGGCGGGGGGGGGGGGGGGCGGGGGGGGGGGGGGGGG@G&lt;GGGGGGGGGGGGFGGFGFF:DGGFFCGGGGCGGFEGGGGGGEGGGFB:GGGDG=FGGGG,GAGGGDGGFGGGCD3GFG+G?G*FGGG&lt;GCGGGGC@CD&lt;FFGF*:+G=FE&lt;;GG,8C7GF=;9D=G&lt;C5C66G6C&gt;7)FD*6G7D)9;*C:)4D5+9&gt;@G*=0):)*;1*504\n']
  &gt;&gt;&gt; r2
  ['@read5000/2\n', 'TGTGGGGTTCTGGATCCCCTCTCCACTTAGGGACTGGGGAACTCCTTAGCTAAGCTAATTACCCCTGGATCAAAATTGGGGCCGCATTGTTCTCTGTCCTGCACAGTTCCCTGCTCGAATCGTTGCTTCCTCTGCCAGGATCCTGGCTTTTCCTACCAGAGAATGTTGAGGTTCTCTGTCATTTTCGCTGCTAATGTGTCTTCACTAATCGTGTTTCCGGGAGACTTGCCACCCCTAAAGGAGCTTGTTGAACACTTGTCACCAGACCTAAGTCCTTAGGGATGAACCTGTGTCTCGGCAC\n', '+\n', 'CCCCBGC9FGCD-GGGFGGGEFEFF@CCGFGGGGFF+DG6,F69+FGG+FGF&lt;FCC;GGEBFGGEDECFGGG6GDG7BFG9GCG9G9GDGGGG&lt;GDFFFGEGC,&lt;GC=GFG:,:A:EE+GD&lt;:E,FGF=FCAGEFF++G,G,&lt;EGGFGFFA&gt;*FG9EGFCFCGC,B&gt;43GFF@E@2&gt;9ED@,568D=A;C+=5@E/F53D&gt;,6F*9@+,E,C=*0+*@56&lt;*9@:/F,1,:*)*14*0CC=*1)*0*,*)0;+)*))/**)**;0;//9)1+)10.()5/)070*)7*C**3/)(2*****\n']
  </code></pre>
  
Tags: <python><fastq>
Title: Why does this naive Python paired Fastq reader work?
ThreadText: # Why does this naive Python paired Fastq reader work?
  
  (Asked by: username_0 on 2020-02-04)
  
  I wrote some throwaway code for ingesting paired-end reads from a Fastq file into memory.
  
  ```py
  def ingest(stream1, stream2):
      while True:
          record1, record2 = list(), list()
          for _ in range(4):
              record1.append(next(stream1))
              record2.append(next(stream2))
          yield record1, record2
  
  ```
  
  I had initially put the `for` loop in a `try/Except` block, anticipating I'd need to explicitly break out of the endless `while` loop. However, the code seems to work correctly even after I removed it. I even "corrupted" the final read in one of the paired files, and the `ingest` function performs reasonably (the last read pair is never `yield`ed).
  
  My question is...**why does this code work**, since I never explicitly break out of the loop?
  
  Here's an example of how the code is invoked.
  
  ```py
  >>> with open('seqR1.fastq', 'r') as fh1, open('seqR2.fastq', 'r') as fh2:
  ...     counter = 0
  ...     for r1, r2 in interleave(fh1, fh2):
  ...         counter += 1
  >>> counter
  5000
  >>> r1
  ['@read5000/1\n', 'AGGTCTGGGGACAGGTGTTTTACAGGCTCCTTTAGGGGTGGCAAGTCTCCCGGAAACATGATTAGTGAAGACACGTTAACAGCGAAACTGACAGAGAACCTCAACATTCTCTGGTAGGAAAAGCCAGGATCCTGGCAGAGGAAGCAACGATTCGAGCAGGGAACTGTGCAGGACAGAGAACAATGCGGCCCCAATTTTGATCCAGGGGTAGTTAGCTTAGCTAAGGAGTTCCCCAGTCCCTAAGTGGAGAGGGGATCCAGAACCCTACAATTTATTATTATATCTTAGCGGATATATCTAT\n', '+\n', 'CCCCCGGGEGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@GGGGGGGFAGGGCGGGGGFGGGGGGGGGCGGGGGDGGGGFGGGGGGGGGCGGGGGGGGGGGGGGGCGGGGGGGGGGGGGGGGG@G<GGGGGGGGGGGGFGGFGFF:DGGFFCGGGGCGGFEGGGGGGEGGGFB:GGGDG=FGGGG,GAGGGDGGFGGGCD3GFG+G?G*FGGG<GCGGGGC@CD<FFGF*:+G=FE<;GG,8C7GF=;9D=G<C5C66G6C>7)FD*6G7D)9;*C:)4D5+9>@G*=0):)*;1*504\n']
  >>> r2
  ['@read5000/2\n', 'TGTGGGGTTCTGGATCCCCTCTCCACTTAGGGACTGGGGAACTCCTTAGCTAAGCTAATTACCCCTGGATCAAAATTGGGGCCGCATTGTTCTCTGTCCTGCACAGTTCCCTGCTCGAATCGTTGCTTCCTCTGCCAGGATCCTGGCTTTTCCTACCAGAGAATGTTGAGGTTCTCTGTCATTTTCGCTGCTAATGTGTCTTCACTAATCGTGTTTCCGGGAGACTTGCCACCCCTAAAGGAGCTTGTTGAACACTTGTCACCAGACCTAAGTCCTTAGGGATGAACCTGTGTCTCGGCAC\n', '+\n', 'CCCCBGC9FGCD-GGGFGGGEFEFF@CCGFGGGGFF+DG6,F69+FGG+FGF<FCC;GGEBFGGEDECFGGG6GDG7BFG9GCG9G9GDGGGG<GDFFFGEGC,<GC=GFG:,:A:EE+GD<:E,FGF=FCAGEFF++G,G,<EGGFGFFA>*FG9EGFCFCGC,B>43GFF@E@2>9ED@,568D=A;C+=5@E/F53D>,6F*9@+,E,C=*0+*@56<*9@:/F,1,:*)*14*0CC=*1)*0*,*)0;+)*))/**)**;0;//9)1+)10.()5/)070*)7*C**3/)(2*****\n']
  
  ```
  
  **Tags:** python, fastq
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-02-04. Score: 4 | Accepted answer
  
  Your code only runs prior to Python 3.7, now it raises `RuntimeError: generator raised StopIteration`. This is essentially a bug fix described in [PEP 479 -- Change StopIteration handling inside generators](https://www.python.org/dev/peps/pep-0479/). You can enable the new behaviour from Python 3.5 via `from __future__ import generator_stop`.
  
  Note, it's not related to file-handles per se, you can reproduce it with a smaller example with a simple iterator:
  
  ```
  it = iter([1,2])
  
  def f(it):
      while True:
          yield next(it)
  
  print(list(f(it)))
  
  ```
  
  You should not write code relying on the bug
  
  
  ---
  
  ### Answer by username_1 on 2020-02-04. Score: 2
  
  EDIT: **[Chris\_Rands's answer](https://bioinformatics.stackexchange.com/a/11311/4980) is more authoritative and clarifies that this behavior is essentially a bug.** (I had only tested my example on Python 3.6.8 and hadn't seen PEP 479.)
  
  When `next` runs out of lines, it raises `StopIteration`, and apparently that stops the while loop similarly to calling `break` without you needing to catch the exception. I couldn't see a definitive source on that but you can see it in action if you catch and re-raise the exception:
  
  ```
  def ingest2(stream1, stream2):
      while True:
          try:
              record1, record2 = list(), list()
              for _ in range(4):
                  record1.append(next(stream1))
                  record2.append(next(stream2))
              yield record1, record2
          except StopIteration:
              print("StopIteration raised")
              raise
  
  ```
  
  You can also try to catch it outside the loop, but it doesn't get re-raised at that point. (If you try raising one yourself in a while loop, though, it *doesn't* catch it in the same way, so there must be some extra Python magic going on. I can't see any difference between the two exception objects so I'd be curious what's going on there.) Neat example. You could also make a pair of FASTQ parsers using Biopython's [SeqIO.parse](https://biopython.org/wiki/SeqIO#sequence-input) for more complex/permanent situations if needed.
  
  
  ---
  
  