title: Python memory error for a large data set
body: <p>I want to generate a 'bag of words' matrix containing documents with the corresponding counts for the words in the document. In order to do this I run below code for initialising the bag of words matrix. Unfortunately I receive a memory error after x amounts of documents in the line where I read the document. Is there a better way of doing this, so that I can avoid the memory error? Please be aware that I would like to process a very large amount of documents ~ 2.000.000 with only 8 Gb of RAM.</p>
  
  <pre><code>def __init__(self, paths, words_count, normalize_matrix = False ,trainingset_size = None, validation_set_words_list = None):
      '''
      Open all documents from the given path.
      Initialize the variables needed in order
      to construct the word matrix.
  
      Parameters
      ----------
      paths: paths to the documents.
      words_count: number of words in the bag of words.
      trainingset_size: the proportion of the data that should be set to the training set.
      validation_set_words_list: the attributes for validation.
      '''
  
      print '################ Data Processing Started ################'
  
      self.max_words_matrix = words_count
  
      print '________________ Reading Docs From File System ________________'
      timer = time()
      for folder in paths:
          self.class_names.append(folder.split('/')[len(folder.split('/'))-1])
          print '____ dataprocessing for category '+folder
          if trainingset_size == None:
              docs = os.listdir(folder)
          elif not trainingset_size == None and validation_set_words_list == None:
              docs = os.listdir(folder)[:int(len(os.listdir(folder))*trainingset_size-1)]
          else:
              docs = os.listdir(folder)[int(len(os.listdir(folder))*trainingset_size+1):]
          count = 1
          length = len(docs)
          for doc in docs:
              if doc.endswith('.txt'):
                  d = open(folder+'/'+doc).read()
                  # Append a filtered version of the document to the document list.
                  self.docs_list.append(self.__filter__(d))
                  # Append the name of the document to the list containing document names.
                  self.docs_names.append(doc)
                  # Increase the class indices counter.
                  self.class_indices.append(len(self.class_names)-1)
  
              print 'Processed '+str(count)+' of '+str(length)+' in category '+folder
              count += 1
  </code></pre>