title: LightFM train_interactions shared among train and test sets: This will cause incorrect evaluation, check your data split
body: <p>tl;dr: Working with Yelp Dataset to make a recommendation System but running into <strong>Test interactions matrix and train interactions matrix share 68 interactions. This will cause incorrect evaluation, check your data split.</strong> error when running the following LightFM code.</p>
  
  <pre><code>test_auc = auc_score(model,
                      test,
                      #train_interactions=train, #Unable to run with this line uncommented
                      item_features=sparse_features_matrix,
                      num_threads=NUM_THREADS).mean()
  print('Hybrid test set AUC: %s' % test_auc)
  </code></pre>
  
  <hr>
  
  <p>Full Story: Working with Yelp Dataset to build a recommendation system.</p>
  
  <p>Going off the code provided in example documentation (<a href="https://making.lyst.com/lightfm/docs/examples/hybrid_crossvalidated.html" rel="nofollow noreferrer">https://making.lyst.com/lightfm/docs/examples/hybrid_crossvalidated.html</a>) for Hybrid Collaborative Filtering.</p>
  
  <p>I ran my code the following way:</p>
  
  <pre><code>from sklearn.model_selection import train_test_split
  from lightfm import LightFM
  from scipy import sparse
  from lightfm.evaluation import auc_score
  
  train, test = train_test_split(sparse_Rating_Matrix, test_size=0.25,random_state=4)
  # Set the number of threads; you can increase this
  # if you have more physical cores available.
  NUM_THREADS = 2
  NUM_COMPONENTS = 100
  NUM_EPOCHS = 3
  ITEM_ALPHA = 1e-6
  
  # Define a new model instance
  model = LightFM(loss='warp',
                  item_alpha=ITEM_ALPHA,
                  no_components=NUM_COMPONENTS)
  
  # Fit the hybrid model. Note that this time, we pass
  # in the item features matrix.
  model = model.fit(train,
                  item_features=sparse_features_matrix,
                  epochs=NUM_EPOCHS,
                  num_threads=NUM_THREADS)
  
  # Don't forget the pass in the item features again!
  train_auc = auc_score(model,
                        train,
                        item_features=sparse_features_matrix,
                        num_threads=NUM_THREADS).mean()
  print('Hybrid training set AUC: %s' % train_auc)
  
  test_auc = auc_score(model,
                      test,
                      #train_interactions=train, # Unable to run with this line uncommented
                      item_features=sparse_features_matrix,
                      num_threads=NUM_THREADS).mean()
  print('Hybrid test set AUC: %s' % test_auc)
  </code></pre>
  
  <p>I had 2 problems:</p>
  
  <p>1) Running the line in question uncommented (train_interactions=train) originally yielded <strong>Inconsistent Shape</strong></p>
  
  <p>which was resolved by the following:
  "test" data set was modified by the following block of code to append a block of zeros below it until the dimensions match that of my train data set (per this recommendation: <a href="https://github.com/lyst/lightfm/issues/369" rel="nofollow noreferrer">https://github.com/lyst/lightfm/issues/369</a>):</p>
  
  <pre><code>#Add X users to Test so that the number of rows in Train match Test
  N = train.shape[0] #Rows in Train set
  n,m = test.shape #Rows &amp; columns in Test set
  
  z = np.zeros([(N-n),m]) #Create the necessary rows of zeros with m columns
  test = test.todense() #Temporarily convert Test into a numpy array
  test = np.vstack((test,z)) #Vertically stack Test on top of the blank users
  test = sparse.csr_matrix(test) #Convert back to sparse
  </code></pre>
  
  <p>2) After the shape issue was resolved, I tried to implement "train_interactions=train"</p>
  
  <p>But ran into <strong>Test interactions matrix and train interactions matrix share 68 interactions. This will cause incorrect evaluation, check your data split.</strong></p>
  
  <p>And I"m not sure how to resolve this 2nd issue. Any ideas?</p>
  
  <p>Details:<br>
  -"sparse_features_matrix" is a sparse matrix of {items x categories} where if an item was "Italian" and "Pizza" then the category of "Italian" and "Pizza" would have a value "1" for that item's row ... "0" elsewhere. <br>
  -"sparse_Rating_Matrix" is a sparse matrix of {users x items} containing values of the user's ratings to the restaurant (item).</p>
  
  <p>04/08/2020 Update: <br>
  LightFM has a whole Database() class object that you should use to prep your data set prior to model evaluation. I found a great github post (<a href="https://github.com/lyst/lightfm/issues/494" rel="nofollow noreferrer">https://github.com/lyst/lightfm/issues/494</a>) where user Med-ELOMARI provides an amazing walk through on a small test data set.</p>
  
  <p>When I prepped my data through this method, I was able to add in user_features that I wanted to model (E.g: User_1592 likes "Thai","Mexican","Sushi" cuisines).</p>
  
  <p>Per Turbo's comment, I used LightFM's random_train_test_split method (had originally split my data via sklearn's train_test_split method) and ran the auc_score with the new train/test sets AND the correctly (as far as im aware) prepared model I still run into the same error code:</p>
  
  <p>Input:</p>
  
  <pre><code>%%time
  (train,test) = random_train_test_split(lightfm_interactions,test_percentage=0.25) #LightFM's method to split
  # Don't forget the pass in the item features again!
  train_auc = auc_score(model_users,
                        train,
                        user_features=lightfm_user_features_list,
                        num_threads=NUM_THREADS).mean()
  print('User_feature training set AUC: %s' % train_auc)
  
  test_auc = auc_score(model_users,
                      test,
                      #train_interactions=train, #Still can't get this to function
                      user_features=lightfm_user_features_list,
                      num_threads=NUM_THREADS).mean()
  print('User_feature test set AUC: %s' % test_auc)
  </code></pre>
  
  <p>Output if "train_interactions=train" is used:</p>
  
  <pre><code>ValueError: Test interactions matrix and train interactions matrix share 435 interactions. This will cause incorrect evaluation, check your data split.
  </code></pre>
  
  <p>Good news however is --- by switching from sklearn's train_test_split to LightFM's random_train_test_split my model's AUC score went from 0.49 to 0.96 on training. So I guess it's important to stick with LightFM's methods if available!</p>