How to design cross-validation and testing scheme when N is small? I have a binary classification problem with 60 samples (N=60). 40 are responders (+) and 20 are non-responders (-) to a drug. There will be ~20 measured features (p=20) per sample with which to make predictions. I expect co-linearity between some of the features. If I devote 20 samples to my test set and divide responders and non-responders equally between the training and testing sets, I would get 6-7 responders in the test set. I would perform cross validation to select between LASSO regression models and random forest with a few parameter values for each. Finally, I would train the best model (one standard error rule) on the entire training set, and then apply to the test set.

*

*I understand that N is too small. Is it foolhardy to even attempt classification in this scenario?


*Is it permissible or even advisable to over-represent the smaller class (non-responders) in the test set? Perhaps I randomly select 10 responders and 10 non-responders for the test set. I might have more statistical confidence. With only 6 non-responders in the test set, it might be easier for me to get "lucky" or "unlucky". A bad classifier could accidentally perform well or vice versa. If my classifier were to perform well by some metric, how could I be confident that was not due to chance?


*Should I use leave-one-out cross validation in this scenario? With 40 samples in the training set and 10 non-responders, I could do 5-folds cross validation and randomly select 2 non-responders and 8 responders for each CV fold. But then I am only training on 30 training samples for each split vs. 39 if I do leave-one-out CV.
I haven't actually done this experiment yet. I am trying to get a sense for whether this kind of a project is worth attempting with such a small N, and if so, what problems I may need to be wary of.

Cross validation is not the only way to compare different models. Several model selection criteria, such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) can precisely be used to compare different hypotheses when the number of observations is small and all samples need to be used for training. The main concept behind these selection criteria is to penalize the likelihood of a model with its complexity (i.e. with its number of free parameters).
