What might be the simplest (least flexible, least expressive) model to avoid over-fit? I need to perform a regression on a data set with a huge noise-to-signal ratio. I am not even sure if there is any "signal" in the data (maybe there is only "noise" in the data), meaning that it might be the case that distribution of targets does not depend on my features at all.
So, in this case I need to work with very simple models but I am afraid that even linear regression is to complex for me. For example, if I have 20 features, I do not believe that I can reliably extract 20+1 parameters from my data.
To simplify my model further, I search a linear function that only depends only on one of my 20 features. However, I am not sure that this is the best way to go.
So, my question is, if there a class of even simpler (less flexible, less expressive) models that are suitable for fitting very very noisy data?
My intuition goes as follows. If we have a logical statement like: "A and B and C", we can simplify it to "A and B" and then we can simplify it to "A". At this point we might think that we cannot say less than just "A" but we are wrong! We can say even less, namely "A or K" or even less: "A or K or L or M".

Use ridge regression.  The ridge parameter is a control on the complexity of the model class.  Ridge regression is equivalent to fitting a regression model with a constraint on the norm of the weight vector, where the constraint depends on the value of the ridge parameter.  If you reduce the ridge parameter slightly, this raises the maximum norm of the weights of the model class slightly, so it can realise any linear function that it could previously realise, and also some more linear functions that require a slightly greater norm.  This means ridge regression provides a nested set of model classes of increasing complexity, indexed by the ridge parameter.  That is why it is effective in avoiding over-fitting.
I explain the connection between regularisation (as used in ridge regression) and constrained optimisation in my answer to another question.
