title: using nltk regex example in scikit-learn CountVectorizer
body: <p>I was trying to use an example from the nltk book for a regex pattern inside the CountVectorizer from scikit-learn. I see examples with simple regex but not with something like this:</p>
  
  <pre><code>pattern = r''' (?x)         # set flag to allow verbose regexps 
      ([A-Z]\.)+          # abbreviations (e.g. U.S.A.)
      | \w+(-\w+)*        # words with optional internal hyphens
      | \$?\d+(\.\d+)?%?  # currency &amp; percentages
      | \.\.\.            # ellipses '''
  
  text = 'I love N.Y.C. 100% even with all of its traffic-ridden streets...'
  vectorizer = CountVectorizer(stop_words='english',token_pattern=pattern)
  analyze = vectorizer.build_analyzer()
  analyze(text)
  </code></pre>
  
  <p>This produces:</p>
  
  <pre><code>[(u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'', u''),
   (u'', u'-ridden', u''),
   (u'', u'', u''),
   (u'', u'', u'')]
  </code></pre>
  
  <p>With nltk, I get something entirely different:</p>
  
  <pre><code>nltk.regexp_tokenize(text,pattern)
  </code></pre>
  
  <p>['I',
   'love',
   'N.Y.C.',
   '100',
   'even',
   'with',
   'all',
   'of',
   'its',
   'traffic-ridden',
   'streets',
   '...']</p>
  
  <p>Is there a way to get the skl CountVectorizer to output the same thing? I was hoping to use some of the other handy features that are incorporated in the same function call.</p>