title: How to Prepare Text for Machine Learning pipeline with compiled Regular Expressions and/or List Comprehension?
body: <p>I am trying to prepare text in a Python function for a machine learning pipeline and I am having trouble to get the correct output. So, I want to lowercase all words, replace symbols by spaces, delete symbols and remove stopwords from nltk. I tried all kinds of different approaches from list comprehension to regex pattern matching, but I still can't get it right. Please help out! Here are the necessary imports and the basic function:</p>
  
  <pre><code>import nltk
  nltk.download('stopwords')
  from nltk.corpus import stopwords
  
  import re
  </code></pre>
  
  <p>Here is the function:</p>
  
  <pre><code>REPLACE_BY_SPACE_RE = re.compile('[/(){}\[\]\|@,;]')
  BAD_SYMBOLS_RE = re.compile('[^0-9a-z #+_]')
  STOPWORDS = set(stopwords.words('english'))
  
  def text_prepare(text):
  """
      text: a string
  
      return: modified initial string
  """
  
  lower = text.lower() # lowercase text
  space_replace = REPLACE_BY_SPACE_RE.sub(" ",lower) #replace REPLACE_BY_SPACE_RE symbols by space in text
  nosymb = BAD_SYMBOLS_RE.sub("",space_replace) # delete symbols which are in BAD_SYMBOLS_RE from text
  text = [word for word in nosymb if word not in STOPWORDS] # delete                                                                                                             stopwords from text
  
  return text
  </code></pre>
  
  <p>Here is a test function:</p>
  
  <pre><code>def test_text_prepare():
      examples = ["SQL Server - any equivalent of Excel's CHOOSE function?",
              "How to free c++ memory vector&lt;int&gt; * arr?"]
      answers = ["sql server equivalent excels choose function", 
             "free c++ memory vectorint arr"]
      for ex, ans in zip(examples, answers):
      if text_prepare(ex) != ans:
          return "Wrong answer for the case: '%s'" % ex
  return 'Basic tests are passed.'
  </code></pre>
  
  <p>Here is my test result:</p>
  
  <pre><code>print(test_text_prepare())
  Wrong answer for the case: 'SQL Server - any equivalent of Excel's CHOOSE function?'
  </code></pre>