Train test split based on statistics I would like to know if there is a method of splitting that is not random but based on the distribution of the train and test data samples feauture values.
Currently I am splitting randomly but stratified with sklearns train_test_split function and there are huge differences in in my AUCs based on the random state given. My theory is that my small test set size (20% of dataset = 33 samples) cannot represent general performance.
My training score mostly stays the same and is not fluctuating as much as my test score.
Are there other splitting strategies so my test set is not randomly filled with outliers but is more representative of my dataset or should I just average over a lot of random states to get a general performance?
I already 5 Fold cross validate on 80% of my dataset and the 20% are my holdout test set.


My theory is that my small test set size (20% of dataset = 33 samples) cannot represent general performance.

Not necessarily true.  The problem with small sample sizes is that the variance of the estimate can be quite high, which explains the "huge differences".
If 20% of your data is 33 observations, then you have ~162 observations, which isn't a ton of data.  I would recommend a repeated cross validation to avoid spurious splits giving you optimistic/pessimistic estimates.
