How to design cross-validation and testing scheme when N is small? I have a binary classification problem with 60 samples (N=60). 40 are responders (+) and 20 are non-responders (-) to a drug. There will be ~20 measured features (p=20) per sample with which to make predictions. I expect co-linearity between some of the features. If I devote 20 samples to my test set and divide responders and non-responders equally between the training and testing sets, I would get 6-7 responders in the test set. I would perform cross validation to select between LASSO regression models and random forest with a few parameter values for each. Finally, I would train the best model (one standard error rule) on the entire training set, and then apply to the test set.

*

*I understand that N is too small. Is it foolhardy to even attempt classification in this scenario?


*Is it permissible or even advisable to over-represent the smaller class (non-responders) in the test set? Perhaps I randomly select 10 responders and 10 non-responders for the test set. I might have more statistical confidence. With only 6 non-responders in the test set, it might be easier for me to get "lucky" or "unlucky". A bad classifier could accidentally perform well or vice versa. If my classifier were to perform well by some metric, how could I be confident that was not due to chance?


*Should I use leave-one-out cross validation in this scenario? With 40 samples in the training set and 10 non-responders, I could do 5-folds cross validation and randomly select 2 non-responders and 8 responders for each CV fold. But then I am only training on 30 training samples for each split vs. 39 if I do leave-one-out CV.
I haven't actually done this experiment yet. I am trying to get a sense for whether this kind of a project is worth attempting with such a small N, and if so, what problems I may need to be wary of.


I understand that N is too small. Is it foolhardy to even attempt classification in this scenario?

Probably, at least as you propose to proceed.
First, for train/test splits to work well, you need on the order of 20,000 cases. You are very far from that. Your best option is to repeat your entire modeling process on multiple bootstrap samples, treat each bootstrap sample as a training set, and evaluate each model against the entire data set as a test set. Incorporating the "optimism bootstrap" will allow you to evaluate the extent of overfitting. That assesses the quality of your modeling process, which is all you can hope to do with a small sample.
For a binary outcome and an unpenalized model, to avoid overfitting you typically need about 15 members of the minority class per predictor. See Chapter 4 of Frank Harrell's course notes. With 20 in your minority class, that's only 1 or maybe 2 unpenalized predictors.
Penalization via LASSO will tend to retain a similar number of predictors, 1 or 2 in this data set. The particular predictors retained, however, are likely to differ from bootstrap sample to bootstrap sample, particularly with collinear predictors. See Statistical Learning with Sparsity, Section 6.2. There's no assurance that those retained in the model on the full data set will be the "best choices" overall, only that they happened to work on this small data set. Is that OK for your application?
Critically, don't use accuracy as your performance criterion. You should work with probability estimates for as long as possible, using log loss or other strictly proper scoring rules. If you do ultimately have to classify, the choice of probability cutoff should take into account the relative costs of false-positive and false-negative classifications. The (often hidden) default of a cutoff at probability of 0.5 for "accuracy" is only appropriate if those are equally costly, which isn't necessarily true.
As you are still in the study design phase, simulate large numbers of cases based on reasonable assumptions about outcomes, their associations with your 20 features, and the correlations among feature values. See how well you can model your known associations with outcome on samples of size 60. That might be the best way to see whether this modeling attempt really is foolhardy.
You might be better off using you knowledge of the subject matter to do data reduction on the set of features, combining them intelligently into a smaller number of combined predictors, rather than relying on the types of automated procedures you envision. Again, see Harrell's course notes for guidance.
