What might be the simplest (least flexible, least expressive) model to avoid over-fit? I need to perform a regression on a data set with a huge noise-to-signal ratio. I am not even sure if there is any "signal" in the data (maybe there is only "noise" in the data), meaning that it might be the case that distribution of targets does not depend on my features at all.
So, in this case I need to work with very simple models but I am afraid that even linear regression is to complex for me. For example, if I have 20 features, I do not believe that I can reliably extract 20+1 parameters from my data.
To simplify my model further, I search a linear function that only depends only on one of my 20 features. However, I am not sure that this is the best way to go.
So, my question is, if there a class of even simpler (less flexible, less expressive) models that are suitable for fitting very very noisy data?
My intuition goes as follows. If we have a logical statement like: "A and B and C", we can simplify it to "A and B" and then we can simplify it to "A". At this point we might think that we cannot say less than just "A" but we are wrong! We can say even less, namely "A or K" or even less: "A or K or L or M".

No overfitting

I search a linear function that only depends only on one of my 20 features

There is not much overfitting when you search for only a linear function
$$y_i = a + b x_{i}$$
Overfitting is not a problem in this case because you do not have hyperparameters that you can tune.
Your problem is more straightforward and only about picking the best feature.
The question of overfitting, and the bias-variance tradeoff, is not relevant here because there is no bias that can make the model less flexible and less variable (except for such techniques as shrinking or Bayesian modeling which could also constrain a one-dimensional model).

Overfitting would be more worrisome if you would be fitting more complex functions of your features (e.g. fit polynomial models). In that case, the hyperparameter would be the order of the polynomial and you can change this.
Or, overfitting could occur if you would allow the use of multiple features, in which case the hyperparameter is in the number of features that you include in the model. (but your case seems to have been restricted to only one single feature)

So in your case,
You can simply perform the fitting repeatedly with different features $x_i$ and select whichever is the best-performing feature (for this you need to have some measure of performance, and ideally also have an idea about the distribution in order to estimate the significance, or use some part of the data to estimate the significance).
Whether this is a good approach is a different question, but if this is the constraint of your question (find a linear function with only one single feature) then there is not much better solution possible than simply selecting the feature that fits your data the best.
