title: Python: Speed up re.sub to redact files (multi-processing doesn't help)
body: <p>I need to redact certain words from several thousand files. There's a separate reference file of about 50,000 words that need to be redacted from the files. </p>
  
  <p>With the code I've written, the process is going to take weeks, and I need to make it faster.</p>
  
  <pre><code>import glob, re, sys
  from multiprocessing.dummy import Pool as ThreadPool
  
  def redact_file(file):
      with open(file, 'r') as myfile:
          data=myfile.read()
  
          for word in words_to_redact:
              search_term = r"(?:\b)"+word+r"(?:\b)"
              data = re.sub(search_term, '#', data, flags=re.IGNORECASE)  #this seems to be the slow bit?
  
      with open(file+'_REDACTED', 'w') as file:
          file.write(data)
  
  
  if __name__ == "__main__":
      words_to_redact = []
      with open ("words_to_redact.txt") as myfile:    #about 50,000 rows in this reference file
          words_to_redact=myfile.read().splitlines()
  
      input_files = glob.glob("input_*.txt")
  
      pool = ThreadPool(multiprocessing.cpu_count()) 
      pool.map(redact_file, input_files)
  </code></pre>
  
  <p>Using multi-processing doesn't seem to help.  </p>
  
  <p>I think the performance issue comes from calling re.sub 50,000 times for each file.  Because this creates a new copy of the "data" string every iteration, I think the process becomes limited by memory/cache speed.</p>
  
  <p>I think I have to use re.sub because using regEx is the only way to match words. </p>
  
  <p>Is there a way to do re.sub without a copy each time, or some other way to make this faster?</p>