title: How can I reference additional fields from a Pandas DataFrame when vectorizing text documents in scikit-learn?
body: <p>I'm building a supervised learning application using scikit-learn. My input data comes from a table. The text document is essentially one column ('description') of this table, but I think I can improve my accuracy by referencing other columns in the vectorization (for example to remove street addresses from the description field).</p>
  
  <p>I think I can do this by specifying my own preprocessor and tokenizer functions when I construct the vectorizer. However, I run into trouble with the vectorizer's fit() method. I'm trying to pass a DataFrame containing my columns as the raw_documents.</p>
  
  <p>When the raw_documents get to CountVectorizer._count_vocab() method to build the vocabulary, the code iterates through each record using "for doc in raw_documents:". I was expecting this to walk through each row in the DataFrame and provide a Pandas Series containing that record as the "doc". This "doc" would then get passed to the analyzer and then to my preprocessor and tokenizer where I could reference the associated fields in the Series by name.</p>
  
  <p>Unfortunately, the default behavior for DataFrame is that <strong>iter</strong>() iterates along the information axis instead of the index axis. This means my vectorizer is now walking along the list of column headings instead of each record row (as a Pandas Series). The data that gets to the analyzer as the "doc" is just the column heading strings.</p>
  
  <p>This simple example shows what I am trying to do. the preprocessor is dumb and incomplete, but shows how I am trying to access the adjacent field on a record. (I could also use direction on how to properly update the description value on the input to avoid the Pandas "SettingWithCopyWarning" problem. I tried to follow the recommendation to use .loc[], but still get the warning.) </p>
  
  <pre><code>import re
  from io import StringIO
  import pandas as pd
  from sklearn.feature_extraction.text import CountVectorizer
  
  def my_preprocessor(record):
      try:
          description = record.loc['description']
          # try to update the description field on this record by removing the street address
          description = re.sub(record['street'], '', description.lower())
          # need help here with SettingWithCopyWarning                           
          record.loc['description'] = description
          return record
      except:
          return record.lower()
  
  data = StringIO('''"id","street","description","label_1","label_2"
      "2341324","123 Elm Street","Pine Point was situated at 123 Elm Street in Boston.",1,1''')
  
  df = pd.read_csv(data)
  
  vect = CountVectorizer(preprocessor=my_preprocessor)
  vect.fit_transform(df)
  print(vect.vocabulary_)
  </code></pre>
  
  <p>Results in the column headings as my vocabulary.</p>
  
  <p>{'id': 1, 'street': 4, 'description': 0, 'label_1': 2, 'label_2': 3}</p>
  
  <p>I looked at a couple of options:</p>
  
  <ul>
  <li>Wrap my input data in a DataFrame subclass (RowIterableDataFrame) that overrides <strong>iter</strong>() with a row-wise iterator implementation. I can make the iterator work, but scikit-learn GridSearchCV does a bunch of slicing of the input data so the RowIterableDataFrame I pass in has been sliced into a subset of data rows as a regular DataFrame again by the time it gets into the _count_vocab() method with the "for doc in raw_documents:"</li>
  <li>Pass in the records using DataFrame's iterrows() or itertuples() methods. This gets the right data in on a row-by-row basis, but fails the check_consistent_length() test when the fit() methods call indexable().</li>
  <li>Subclass CountVectorizer and write my own version of the _count_vocab() method that iterates through raw_documents differently in the case of DataFrames (i.e. using .iloc[] indexing). I'd rather not do this because _count_vocab() does a bunch of other stuff I don't want to risk breaking.</li>
  <li>Pre-process my records outside of scikit learn to build a delimited string as input. Pass a list of these in as the raw_documents and then parse them in my preprocessor. This means extra passes through the data.</li>
  <li>Pass in the records using DataFrame.to_dict(orient='records'). This gets me the right data on a row-by-row basis and keeps the column indices for referencing by my preprocessor and tokenizer. The downside appears to be that I have to copy out the data for each row into that dictionary instead of referencing the original data in the DataFrame as a Series.</li>
  </ul>
  
  <p>I would welcome some guidance on how to do this. Perhaps changing the iteration behavior of a Pandas DataFrame or the simplest approach to extending the CountVectorizer.</p>