title: Looking for a quick way to speed up my code
body: <p>I am looking for a way to speed up my code. I managed to speed up most parts of my code, reducing runtime to about 10 hours, but it's still not fast enough and since I'm running out of time I'm looking for a quick way to optimize my code. </p>
  
  <p><strong>An example:</strong></p>
  
  <pre><code>text = pd.read_csv(os.path.join(dir,"text.csv"),chunksize = 5000)
  new_text = [np.array(chunk)[:,2] for chunk in text]
  new_text = list(itertools.chain.from_iterable(new_text))
  </code></pre>
  
  <p>In the code above I read in about 6 million rows of text documents in chunks and flatten them. This code takes about 3-4 hours to execute. This is the main bottleneck of my program. <strong>edit</strong>: I realized that I wasn't very clear on what the main issue was, <strong>The flattening is the part which takes the most amount of time.</strong></p>
  
  <p>Also this part of my program takes a long time:</p>
  
  <pre><code>    train_dict = dict(izip(text,labels))
      result = [train_dict[test[sample]] if test[sample] in train_dict else predictions[sample] for sample in xrange(len(predictions))]
  </code></pre>
  
  <p>The code above first zips the text documents with their corresponding labels (This a machine learning task, with the train_dict being the training set). Earlier in the program I generated predictions on a test set. There are duplicates between my train and test set so I need to find those duplicates. Therefore, I need to iterate over my test set row by row (2 million rows in total), when I find a duplicate I actually don't want to use the predicted label, but the label from the duplicate in the train_dict. I assign the result of this iteration to the variable result in the above code.</p>
  
  <p>I heard there are various libraries in python that could speed up parts of your code, but I don't know which of those could do the job and right know I do not have the time to investigate this, that is why I need someone to point me in the right direction. Is there a way with which I could speed the code snippets above up? </p>
  
  <p><strong>edit2</strong></p>
  
  <p>I have investigated again. And it is definitely a memory issue. I tried to read the file in a row by row manner and after a while the speed declined dramatically, furthermore my ram usage is nearly 100%, and python's disk usage increased sharply. How can I  decrease the memory footprint? Or should I find a way to make sure that I don't hold everything into memory?</p>
  
  <p><strong>edit3</strong>
  As memory is the main issue of my problems I'll give an outline of a part of my program. I have dropped the predictions for the time being, which reduced the complexity of my program significantly, instead I insert a standard sample for every non duplicate in my test set.</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  import itertools
  import os
  
  train = pd.read_csv(os.path.join(dir,"Train.csv"),chunksize = 5000)
  train_2 = pd.read_csv(os.path.join(dir,"Train.csv"),chunksize = 5000)
  test = pd.read_csv(os.path.join(dir,"Test.csv"), chunksize = 80000)
  sample = list(np.array(pd.read_csv(os.path.join(dir,"Samples.csv"))[:,2]))#this file is only 70mb
  sample = sample[1]
  test_set = [np.array(chunk)[:,2] for chunk in test]
  test_set = list(itertools.chain.from_iterable(test_set))
  
  train_set = [np.array(chunk)[:,2] for chunk in train]
  train_set = list(itertools.chain.from_iterable(train_set))
  labels = [np.array(chunk)[:,3] for chunk in train_2]
  labels = list(itertools.chain.from_iterable(labels))
  
  """zipping train and labels"""
  train_dict = dict(izip(train,labels))
  """finding duplicates"""
  results = [train_dict[test[item]] if test[item] in train_dict else sample for item in xrange(len(test))]
  </code></pre>
  
  <p>Although this isn't my entire program, this is the part of my code that needs optimization. As you can see I am only using three important modules in this part, pandas, numpy and itertools. The memory issues arise when flattening train_set and test_set. The only thing I am doing is reading in the files, getting the necessary parts zipping the train documents with the corresponding labels in a dictionary. And then search for duplicates.</p>
  
  <p><strong>edit 4</strong>
  As requested I'll give an explanation of my data sets. My Train.csv contains 4 columns. The first columns contain ID's for every sample, the second column contains titles and the third column contains text body samples(varying from 100-700 words). The fourth column contains category labels. Test.csv contains only the ID's and text bodies and titles. The columns are separated by commas.</p>