title: iterating a text analysis over observations grouped by
body: <p>I have a database with job postings from different sources. Some observations are duplicates but the job description may slightly differ. And I want to identify them by doing text similarity analysis so then I can eliminate those with more than 95% of similarity with other entry.
  My problem is that I do not want the similarity analysis to be done 1to1 over all the text descriptions, but only those with the same company and title. I thought of inserting a group_key, but then I do not know how to make the analysis run &quot;by&quot; each key. Can somebody help?</p>
  <p>My code goes like this:</p>
  <pre><code>df = pd.read_csv(INPUT_DIR +  r'test.csv', usecols=['description', 'company', 'title'])
  df[&quot;id&quot;] = df.index + 1
  
  # I generate the token, apply Tfidvector and compute Cosine Similarity.
  def tokenize(text):
      tokens = nltk.word_tokenize(text)
      return tokens
  vectorizer = TfidfVectorizer(tokenizer=tokenize)
  vector_list = vectorizer.fit_transform(df['clean_text'])
  print(vectorizer.get_feature_names())
  similarity_score = cosine_similarity(vector_list)
  
  print(vector_list.shape)
  print(similarity_score)
  
  # Initiate an empty list to store results.
  results_list = []
  
  # Set a similarity threshold
  similarity_threshold = 0.95
  for i in range(len(similarity_score)):
      for j in range(len(similarity_score[i])):
          if 1 &gt; similarity_score[i][j] &gt;= similarity_threshold:
              results_list.append([df['id'][i], df['group_key'][i], df['title'][i], df['company'][i],df['job_description'][i],df['location_path3'][i], df['id'][j], similarity_score[i][j]])
          else:
              pass
  
  #store results
  results_df = pd.DataFrame(results_list)
  if len(results_df) != 0:
      results_df.columns = ['id','title','company','description','id match', 'similarity_score']
  else:
      pass
  print(results_df)
  results_df.to_csv(INPUT_DIR + 'results_r_df.csv')
  
  </code></pre>