title: Python filter massive iterator with multiprocessing
body: <p>Here is my code before using multiprocessing. It is a task to get the number of items in a massive iterator which satisfy specified conditions:</p>
  <pre><code>from itertools import permutations
  
  def f(input_):
      if 'AB' in ''.join(input_):
          return True
      else：
          return False
  
  if __name__ == &quot;__main__&quot;:
      iterator = permutations(['A', 'B',...])
      count = 0
      for item in iterator:  # it is an itertools.permutations object, with str inside
          if f(item):
              count += 1
      print(count)
  </code></pre>
  <p>But the iterator is too massive that I need to do multiprocessing or multithread(not sure which is better) to speed the process up.</p>
  <p>I have referred to many online references about multi task in Python, and I have tried a few ways. Unfortunately, I still cannot find a solution, since each method I tried is having some problems.
  For example:</p>
  <pre><code>from multiprocessing import Pool
  
  def f(input_):
      if 'AB' in ''.join(input_):
          return True
      else：
          return False
  
  if __name__ == &quot;__main__&quot;:
      pool = Pool()
      result = pool.imap_unordered(f, iterator)
      print(sum(result))
  </code></pre>
  <p>In this example, the problem is that this code runs even slower than my original one. I have also tried using pool.map(), but it is also slower than before and it uses up all of my ram.</p>
  <p>How should I do to make this filtering task as fast as possible using all my CPU's ability? Multiprocessing and multithreading really confuse me. :(</p>