title: cross validation and text categorization
body: <p>I have the same question which has been asked in <a href="https://stackoverflow.com/questions/22672436/sklearn-vectorizing-in-cross-validation-for-text-classification">here</a>: </p>
  
  <p>I have a question about using cross validation in text classification in <code>sklearn</code>. It is problematic to vectorize all data before cross validation, because the classifier would have "seen" the vocabulary occurred in the test data. <code>Weka</code> has filtered classifier to solve this problem. What is the <code>sklearn</code> equivalent for this function? I mean for each fold, the feature set would be different because the training data are different.</p>
  
  <p>However, because I am doing lots of processing for the data between the categorization step and the classification step, I cannot use pipelines ... and was trying to implement the cross validation by my self as an outer loop for the whole process ... any guidance on this as I am fairly new to both python and sickitlearn </p>