Cross Validation of Random Forest results in two different score based on when the test set is created
I have a dataset where I dont have label for 70% of data. So, my target is to train a model on 30% data that has label and then apply this model to get the label for the rest 70% of data. But, it is highly imbalanced and no single model performs well on this. 392
So, I took the following approach :
Method 1 ````python missing_df = df[df.label.isna()] nomissing_df = df[~df.label.isna()]
split the labeled dataset into 80-20 parts for train and validation
...... = train_test_split(nomissing_df)
while new samples are added : #train a 5 Fold Random Forest on the train set clf = RandomForest(....) # code for training
#evaluate the missing data label = clf.predict_proba(missing_df)
if label > 0.75 for all fold : add to positive training samples if label < 0.25 for all fold : add to negative training samples
**Method 2**python missing_df = df[df.label.isna()] nomissing_df = df[~df.label.isna()]
NOOOOOOOOOOOOOOOOOOOOOOOO SPLIT HERE
while new samples are added : #train a 5 Fold Random Forest on the train set clf = RandomForest(....) # code for training
#evaluate the missing data label = clf.predict_proba(missing_df)
if label > 0.75 for all fold : add to positive training samples if label < 0.25 for all fold : add to negative training samples
split the full label dataset ( original + by the above process)
...... = train_test_split(final_train_df)
train final model
clf = RandomForest().train(....)
evaluate on validation set
```` Result
As our dataset is highly skewed, I used precision as my evaluation metric. For method 1, training precision is around 80% but when evaluated at the held-out validation set (at the beginning), the score is 33%.
But for method 2, the precision score for both train and validation is 80%.
My question is why is a discrepancy in validation accuracy? Is method 2 is wrong or any chance of data leakage here? Or any suggestion to improve the precision of method 1 as I think method 1 more correct way to do this? Finally, is this approach of labeling data is correct or there is any other way to do this better? 392