Answers.Body: <p>I'd start by saying that Python is inherently slow and this feels like a case of perhaps unnecessary optimisation. With that in mind here are my suggested (untested) optimisations of @gaspanic code:</p>
  <pre><code>pos = {2, 5, 10, 11, 14} # make pos a set for O(1) member testing, and don't define inside the loop
  for seq in (seqE3,seqE4,seqE5): # Define seq here rather than inside the loop, and use an immutable tuple instead of a mutable list  
      new_seq = []
      count = 0
      for s in seq:
          if s == &quot;-&quot;:
              new_seq.append(s)
          else:
              count += 1
              new_seq.append('Z' if count in pos else s) # use ternary operator
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T11:42:54.017Z
Answers.Text: Thanks, the tuple did add speed, particularly tuple the iterator output. Using a `set` (the `{}` brackets) was a good solution and reduced runtime by ~0.5 seconds, which I'd not considered. It was better than `tuple` in this example. Repositioning `pos` adds a tiny amount, but agree its better practice and depends on the size of `pos` (in this case its small). Happy to discuss `Python`'s slow speed in a summary.
Answers.UserDisplayName: M__
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-03T21:34:59.983Z
Answers.LastActivityDate: 2022-08-03T21:34:59.983Z
Answers.UserDisplayName: Chris_Rands
Answers.Body: <p>Huge thanks to @Chris_Rands, this solution increases runtime by ~ 0.5 seconds uniformly across all coded solutions, so this code has a runtime of just over 3 seconds runtime on the benchmark machine. I incorporated my solution with @Chris_Rands solution and the final runtime I got was:</p>
  <p>@gaspanic</p>
  <blockquote>
  <p>1.7 - 1.8 seconds</p>
  </blockquote>
  <p>This being twice the fastest speed in the question and four times the speed of my raw code.</p>
  <p>My code</p>
  <blockquote>
  <p>4.4 - 4.5 seconds</p>
  </blockquote>
  <p>However a unique combination is needed:</p>
  <ol>
  <li><code>tuple</code> (Chris Rands)</li>
  <li><code>set</code> - important (Chris Rands)</li>
  <li>an <code>iterator</code></li>
  <li>a <code>list comprehension</code></li>
  </ol>
  <p>In particular 3. and 4. are needed in combination. This is explained below.</p>
  <pre><code>def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return tuple(''.join(list(repeat(seq, x))))
  
  def mIterit(seqE3,seqE4,seqE5):
      pos = {2, 5, 10, 11, 14}
      for seqbit in (seqE3,seqE4,seqE5):
          seq = seqbit
          count = 0
          # loc = 0
          for s in seq:
              if s == &quot;-&quot;:
                  yield '-'
              else:
                  count += 1
                  if count in pos:
                      yield 'Z'
                  else:
                      yield s
  
  f __name__ == '__main__':
      
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
      
      listComp = [i for i in mIterit(seqE3,seqE4,seqE5)]
      
  </code></pre>
  <p>The list comprehension (variable <code>listComp</code>) is the very last line and is crucial to the solution otherwise there is little increase in speed to @Chris_Rand's code. This alternative is to replace the last line with:</p>
  <pre><code>newseq = list()
  for i in mIterit(seqE3,seqE4,seqE5):
      newseq.append(i)   
  </code></pre>
  <p>A simple:</p>
  <pre><code>for i in mIterit(seqE3,seqE4,seqE5):
      pass
  </code></pre>
  <p>Is the same speed as the list comprehension, but its of no value because the output isn't saved to anything.</p>
  <p>I was impressed by the interaction between the <code>iterator</code> and <code>list comprehension</code>. The other notable influence on speed is the <code>set</code> described by @Chris_Rand.</p>
  <p>How does it work? <code>tuples</code> do increase speed by removing some functional of a <code>list</code> (immutability). Here the gain is small. Placing the <code>pos</code> variable outside the loop speaks for itself (0.1 seconds gain). <code>set</code> is interesting.</p>
  <p>The <code>iterator</code> spews out a stream of results via <code>yield</code> which is where it attains its speed. The problem is capturing that stream outside the <code>iterator</code> is expensive. However <code>list comprehensions</code> are faster than standard loops and in this scenerio carry little/no detectable overhead. I was impressed. Up to this point I thought <code>list comprehensions</code> were a bit faster and a lot 'cooler' than loops. I hadn't understood there was a synergy with an <code>iterator</code>.</p>
  <p>Is it all worth it? <code>Python</code> can easily achieve faster run times than <code>Java</code> using Cython but I will place the discussion in the <em>chat</em> forum next week.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-05T02:28:03.897Z
Answers.LastActivityDate: 2022-08-05T05:29:05.230Z
Answers.LastEditDate: 2022-08-05T05:29:05.230Z
Answers.UserDisplayName: M__
Body: <p>I discussed a question with @gaspanic <a href="https://bioinformatics.stackexchange.com/questions/14625/python-biopython-replace-amino-acid-residue-on-msa-with-z-from-a-list-of-una">Python/Biopython - Replace amino acid residue on MSA with &quot;z&quot; from a list of unaligned positions</a>
  . The issue emerged was speeding up lists in Python. The solutions we discussed were</p>
  <ol>
  <li>Raw code</li>
  <li>Predefining a list size</li>
  <li><code>deque</code></li>
  </ol>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: left;">1. @gaspanic</th>
  <th style="text-align: left;">my code</th>
  <th style="text-align: left;">2. @gaspanic</th>
  <th style="text-align: left;">my code</th>
  <th style="text-align: left;">3. @gaspanic</th>
  <th style="text-align: left;">my code</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: left;">3.45-3.48</td>
  <td style="text-align: left;">7.17-7.2</td>
  <td style="text-align: left;">4.38-4.85</td>
  <td style="text-align: left;">7.09-7.23</td>
  <td style="text-align: left;">3.44-3.53</td>
  <td style="text-align: left;">7.21-7.39</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Run timings are seconds (3 replicated) above with @gaspanic's code being faster.</p>
  <p>What improvements would speed this up?</p>
  <p>Here I extended the data set</p>
  <pre><code>from itertools import repeat
  def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return ''.join(list(repeat(seq, x)))
  
  if __name__ == '__main__':
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
  </code></pre>
  <p>@gaspanic code</p>
  <pre><code>for seqbit in [seqE3,seqE4,seqE5]:
      seq = seqbit
      pos = [2, 5, 10, 11, 14]    
      new_seq = []
      count = 0
      for s in seq:
          if s == &quot;-&quot;:
              new_seq.append(s)
          else:
              count += 1
              if count in pos:
                  new_seq.append('Z')
              else:
                  new_seq.append(s)
     # print(''.join(new_seq))
  </code></pre>
  <p>My code:</p>
  <pre><code># same variables and loop structure
  newseq = list()
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq.append(s)
  # print(''.join(newseq))
  </code></pre>
  <p>Predefining lists code (additional counter needed for @gaspanic's code:</p>
  <pre><code># same variables and loop structure
  new_seq = [] * len(seq)
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq[n] = s
  </code></pre>
  <p>Deque code simply any of the above substituting</p>
  <pre><code>newseq = list() 
  </code></pre>
  <p>with</p>
  <pre><code>from collections import deque 
  newseq = deque()
  </code></pre>
  <p>Suggestions for efficiency improvements welcome.</p>
  <p>A solution in <code>Cython</code> code would be cool. <code>PyPI</code> is not sought here.</p>
  
Tags: <python><phylogenetics><phylogeny><algorithms><benchmarking>
Title: Improving list speed: glycosylation example
ThreadText: # Improving list speed: glycosylation example
  
  (Asked by: username_0 on 2022-08-03)
  
  I discussed a question with @gaspanic [Python/Biopython - Replace amino acid residue on MSA with "z" from a list of unaligned positions](https://bioinformatics.stackexchange.com/questions/14625/python-biopython-replace-amino-acid-residue-on-msa-with-z-from-a-list-of-una)
  . The issue emerged was speeding up lists in Python. The solutions we discussed were
  
  1. Raw code
  2. Predefining a list size
  3. `deque`
  
  | 1. @gaspanic | my code | 2. @gaspanic | my code | 3. @gaspanic | my code |
  | --- | --- | --- | --- | --- | --- |
  | 3.45-3.48 | 7.17-7.2 | 4.38-4.85 | 7.09-7.23 | 3.44-3.53 | 7.21-7.39 |
  
  Run timings are seconds (3 replicated) above with @gaspanic's code being faster.
  
  What improvements would speed this up?
  
  Here I extended the data set
  
  ```
  from itertools import repeat
  def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return ''.join(list(repeat(seq, x)))
  
  if __name__ == '__main__':
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
  
  ```
  
  @gaspanic code
  
  ```
  for seqbit in [seqE3,seqE4,seqE5]:
      seq = seqbit
      pos = [2, 5, 10, 11, 14]    
      new_seq = []
      count = 0
      for s in seq:
          if s == "-":
              new_seq.append(s)
          else:
              count += 1
              if count in pos:
                  new_seq.append('Z')
              else:
                  new_seq.append(s)
     # print(''.join(new_seq))
  
  ```
  
  My code:
  
  ```
  # same variables and loop structure
  newseq = list()
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq.append(s)
  # print(''.join(newseq))
  
  ```
  
  Predefining lists code (additional counter needed for @gaspanic's code:
  
  ```
  # same variables and loop structure
  new_seq = [] * len(seq)
  count = 0
  for n,s in enumerate(prot):    
      if str(s) == '-':
          count += 1
          s = '-'
      else:
          if (n + 1 - count) in pos:
              s = 'Z'
      newseq[n] = s
  
  ```
  
  Deque code simply any of the above substituting
  
  ```
  newseq = list() 
  
  ```
  
  with
  
  ```
  from collections import deque 
  newseq = deque()
  
  ```
  
  Suggestions for efficiency improvements welcome.
  
  A solution in `Cython` code would be cool. `PyPI` is not sought here.
  
  **Tags:** python, phylogenetics, phylogeny, algorithms, benchmarking
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-08-03. Score: 2 | Accepted answer
  
  I'd start by saying that Python is inherently slow and this feels like a case of perhaps unnecessary optimisation. With that in mind here are my suggested (untested) optimisations of @gaspanic code:
  
  ```
  pos = {2, 5, 10, 11, 14} # make pos a set for O(1) member testing, and don't define inside the loop
  for seq in (seqE3,seqE4,seqE5): # Define seq here rather than inside the loop, and use an immutable tuple instead of a mutable list  
      new_seq = []
      count = 0
      for s in seq:
          if s == "-":
              new_seq.append(s)
          else:
              count += 1
              new_seq.append('Z' if count in pos else s) # use ternary operator
  
  ```
  
  > **Comments:**
  > **username_0**: Thanks, the tuple did add speed, particularly tuple the iterator output. Using a `set` (the `{}` brackets) was a good solution and reduced runtime by ~0.5 seconds, which I'd not considered. It was better than `tuple` in this example. Repositioning `pos` adds a tiny amount, but agree its better practice and depends on the size of `pos` (in this case its small). Happy to discuss `Python`'s slow speed in a summary.  
  
  ---
  
  ### Answer by username_0 on 2022-08-05. Score: 1
  
  Huge thanks to @Chris\_Rands, this solution increases runtime by ~ 0.5 seconds uniformly across all coded solutions, so this code has a runtime of just over 3 seconds runtime on the benchmark machine. I incorporated my solution with @Chris\_Rands solution and the final runtime I got was:
  
  @gaspanic
  
  > 1.7 - 1.8 seconds
  
  This being twice the fastest speed in the question and four times the speed of my raw code.
  
  My code
  
  > 4.4 - 4.5 seconds
  
  However a unique combination is needed:
  
  1. `tuple` (username_1)
  2. `set` - important (username_1)
  3. an `iterator`
  4. a `list comprehension`
  
  In particular 3. and 4. are needed in combination. This is explained below.
  
  ```
  def iterit(x):
      seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
      return tuple(''.join(list(repeat(seq, x))))
  
  def mIterit(seqE3,seqE4,seqE5):
      pos = {2, 5, 10, 11, 14}
      for seqbit in (seqE3,seqE4,seqE5):
          seq = seqbit
          count = 0
          # loc = 0
          for s in seq:
              if s == "-":
                  yield '-'
              else:
                  count += 1
                  if count in pos:
                      yield 'Z'
                  else:
                      yield s
  
  f __name__ == '__main__':
      
      seqE3 = iterit(2800) # length is 201600
      seqE4 = iterit(28000) # length is 2016000
      seqE5 = iterit(280000) # length is 20160000
      
      listComp = [i for i in mIterit(seqE3,seqE4,seqE5)]
      
  
  ```
  
  The list comprehension (variable `listComp`) is the very last line and is crucial to the solution otherwise there is little increase in speed to @Chris\_Rand's code. This alternative is to replace the last line with:
  
  ```
  newseq = list()
  for i in mIterit(seqE3,seqE4,seqE5):
      newseq.append(i)   
  
  ```
  
  A simple:
  
  ```
  for i in mIterit(seqE3,seqE4,seqE5):
      pass
  
  ```
  
  Is the same speed as the list comprehension, but its of no value because the output isn't saved to anything.
  
  I was impressed by the interaction between the `iterator` and `list comprehension`. The other notable influence on speed is the `set` described by @Chris\_Rand.
  
  How does it work? `tuples` do increase speed by removing some functional of a `list` (immutability). Here the gain is small. Placing the `pos` variable outside the loop speaks for itself (0.1 seconds gain). `set` is interesting.
  
  The `iterator` spews out a stream of results via `yield` which is where it attains its speed. The problem is capturing that stream outside the `iterator` is expensive. However `list comprehensions` are faster than standard loops and in this scenerio carry little/no detectable overhead. I was impressed. Up to this point I thought `list comprehensions` were a bit faster and a lot 'cooler' than loops. I hadn't understood there was a synergy with an `iterator`.
  
  Is it all worth it? `Python` can easily achieve faster run times than `Java` using Cython but I will place the discussion in the *chat* forum next week.
  
  
  ---
  
  