Thousands of features and only 70 samples I am working on a regression problem where I have around 5-10 thousand features and have only 65 samples. I am training my algorithm on 55 samples and testing on 10 samples. I am using both Pearson correlation and feature importance using the random forest to remove irrelevant features. In testing data, the random forest algorithm is predicting the average of the training samples.
I have a few questions:
(1) Which ML algorithm should I use when I have very few samples for training (55 samples in my case)?
(2) The Best way of feature selection when we have much more features than samples?

Others have described your impossible challenge very well.  To state in other ways:

*

*Your sample size is too small by a factor of 100 for split-sample validation to have any hope of working, i.e., of being precise and stable

*Statistical testing to discard features has no hope of discarding the right features

*Penalization (regularization; shrinkage) has no hope here because the sample size is too small for you to be able to choose the shrinkage factor

*Your only hope is to use unsupervised learning (data reduction; does not use the outcome variable in any way) methods such as principal components, variable clustering, sparse principal components to reduce the dimensionality of the feature space down to 1 or 2 numbers per observation, then using those 1 or 2 numbers to predict the outcome.  In other words the only hope of meeting your goal is for you to be lucky enough that the ways the features are redundant with each other can be collapsed into some kind of signal summary that relates to Y.

