Train test split based on statistics I would like to know if there is a method of splitting that is not random but based on the distribution of the train and test data samples feauture values.
Currently I am splitting randomly but stratified with sklearns train_test_split function and there are huge differences in in my AUCs based on the random state given. My theory is that my small test set size (20% of dataset = 33 samples) cannot represent general performance.
My training score mostly stays the same and is not fluctuating as much as my test score.
Are there other splitting strategies so my test set is not randomly filled with outliers but is more representative of my dataset or should I just average over a lot of random states to get a general performance?
I already 5 Fold cross validate on 80% of my dataset and the 20% are my holdout test set.


should I just average over a lot of random states to get a general performance?

You can certainly do that. Repeated (aka iterated) k-fold cross validation allows you to measure the variance for the same case tested by different surrogate models, so you can also derive figures of merit about model stability on test case predictions.

 Are there other splitting strategies so my test set is not randomly filled with outliers but is more representative of my dataset

Yes.

*

*on the label side, you can stratify

*of the feature side, there is a number of algorithms that were developed for a somewhat different sceanario, but which you can adapt: they are to select a smallish number of already measured samples for which (expensive) labels are to be obtained - typically for regression model development, where a more-or-less uniform distribution of labelled samples over feature space is desired.

*

*have a look at R package prospectr, which has a nice summary discussion for a bunch of algorithms

*Kennard-Stones and Duplex algorithms use the heuristic of iteratively adding points that are furthest away from already chosen points

*You can also use k-means clustering. k-means has the (for clustering mostly undesirable) characteristic that the clusters will roughly be spheres of the same size. You can use this by running k-means with the number of clusters being the desired number of test cases and then randomly choosing 1 case per cluster for testing and use the others for training in a set validation. The choice of test case step can be repeated, so in the end you'll also have tested all cases by some surrogate models.
This is called naes() in prosprectr, literature reference is likely T. Naes, T. Isaksson, T. Fearn, T. Davies, (2002) A user friendly guide to multivariate calibration and classification.



*More heuristics: removing close neighbour points (Puchwein, SELECT)

... but of course, an adequate proportion of "outliers" is required to correctly represent your data.
If you believe your "outliers" truly do not belong into your data set, you can give an objective why that is the case (being outliers, i.e. far away from the majority of data points is not a good reason!) and remove them from the data set, or report your model's performance separately for these points.
