What might be the simplest (least flexible, least expressive) model to avoid over-fit? I need to perform a regression on a data set with a huge noise-to-signal ratio. I am not even sure if there is any "signal" in the data (maybe there is only "noise" in the data), meaning that it might be the case that distribution of targets does not depend on my features at all.
So, in this case I need to work with very simple models but I am afraid that even linear regression is to complex for me. For example, if I have 20 features, I do not believe that I can reliably extract 20+1 parameters from my data.
To simplify my model further, I search a linear function that only depends only on one of my 20 features. However, I am not sure that this is the best way to go.
So, my question is, if there a class of even simpler (less flexible, less expressive) models that are suitable for fitting very very noisy data?
My intuition goes as follows. If we have a logical statement like: "A and B and C", we can simplify it to "A and B" and then we can simplify it to "A". At this point we might think that we cannot say less than just "A" but we are wrong! We can say even less, namely "A or K" or even less: "A or K or L or M".

You can try to apply feature selection. Linear models are already the simplest functional form that you can choose, but techniques like LASSO, non-negative garrote and other penalized/restricted estimation methods try to reduce dimensions, keeping only the most "worthy" covariables of the model. Penalized least squares has the following objective function,
$$g(\beta;\lambda) = ||Y-X\beta||_2^2 + \lambda||\beta||_k$$
$$\hat{\beta}_{PLS} = \arg\min_{\beta\in \mathbb{R}^p} g(\beta,\lambda)$$
such that the model becomes the LASSO when $k=1$ or the ridge regression when $k=2$. There are multiple extensions that apply less common penalties or work with different values of $\lambda$ for each variable, that is also a multitude of side-methods to choose the $\lambda$.
If your data matrix is close to singularity, then ridge ($L^2$ penalty) is a good option.
If your interesting in feature selection, then LASSO is the way to go ($L^1$ penalty).
Elastic net is a convex combination between penalties and gives better estimates in some cases, $\alpha||\beta||_1 + (1-\alpha)||\beta||_2^2$.
Another option that is similar to the LASSO is the Dantzig selector and I will leave the article by Terence Tao that gives a pretty good explanation:
https://terrytao.wordpress.com/2008/03/22/the-dantzig-selector-statistical-estimation-when-p-is-much-larger-than-n/
EDIT
As I said in the comments, there's a duality between penalties and restrictions over the parameter space of the form $||\beta||_k \leq C$, I'm adding an image that illustrates the $L^p$-balls for various $p$s,

Shrinkage estimation is a huge research area and there are lots of results that show that penalized estimators often have smaller risk (expected loss) than common procedures, specially in high $p$, low $n$ situations. Just for curiosity, take a look at the James-Stein estimator.
If you want a model with only one covarible as you mentioned, you may raise $\lambda$ incrementally until you get only on $\beta\neq 0$.
Other valid methodology is to apply some kind of screening based on correlations, t-tests or other kind of association measures that rank variables according to relevance before estimation.
