What might be the simplest (least flexible, least expressive) model to avoid over-fit? I need to perform a regression on a data set with a huge noise-to-signal ratio. I am not even sure if there is any "signal" in the data (maybe there is only "noise" in the data), meaning that it might be the case that distribution of targets does not depend on my features at all.
So, in this case I need to work with very simple models but I am afraid that even linear regression is to complex for me. For example, if I have 20 features, I do not believe that I can reliably extract 20+1 parameters from my data.
To simplify my model further, I search a linear function that only depends only on one of my 20 features. However, I am not sure that this is the best way to go.
So, my question is, if there a class of even simpler (less flexible, less expressive) models that are suitable for fitting very very noisy data?
My intuition goes as follows. If we have a logical statement like: "A and B and C", we can simplify it to "A and B" and then we can simplify it to "A". At this point we might think that we cannot say less than just "A" but we are wrong! We can say even less, namely "A or K" or even less: "A or K or L or M".

I will assume that you want to avoid overfitting so that your model is the most predictive on new data, rather than being overfit to the training data. Rather than constraining the class of models you are willing to consider at the outset, you may want to fit multiple models and then choose the model that "performs the best" on out-of-sample data. You will need some loss function to assess which "performs the best" out of sample, such as mean squared error. You can estimate how well your model performs on new data by splitting into training and test sets, by using cross validation, or related methods. Candidate models might include those already mentioned, such as ordinary regression, LASSO regression, ridge regression, elastic net etc. but you could also try others. Custom software in R, such as the glmnet package, should make it relatively easy to do a lot of the above.
