Distributional Assumptions in Machine Learning In more classical statistical methods like linear regression, we can quantify how well our model generalizes under certain strong assumptions. For example, we know that $\hat Y = X \hat \beta \sim \mathcal{N}(X\beta, \sigma^2 H)$ assuming that $Y\sim \mathcal{N}(X \beta, \sigma^2)$.
In machine learning theory, the approach is completely different: We do not make any assumptions on the (conditional) distribution and use tools like VC Dimensions to derive bounds on the difference $\sup_{h \in \mathcal{H}} |L_S(h) - L(h)|$ for a given function class $\mathcal{H}$ with $L_S(h)$ and $L(h)$ denoting the empirical and the "true" error of a hypothesis $h \in \mathcal{H}$. This then allows us to bound $L(\hat h)$ where $\hat h$ is our fitted model. These bounds are obviously very loose.
My question: Is there a theory bridging this gap between "no assumptions, mostly loose bounds" and "strong assumptions, very good bounds"?
I am aware of no free lunch. However, in most applications of machine learning we have real-world distributions  (whatever that exactly means) that seem to be quite nicely behaved in the sense that our model generalizes better than we would necessarily expect from learning theory.

Mathematically rich question. VC dimension build on top of, similarly PAC complexity, on binary outcomes. There are generalisations to VC.

Is there a theory bridging this gap between "no assumptions, mostly loose bounds" and "strong assumptions, very good bounds"?

Yes. There are propositions from the literature. Probably the most prominent is the Rademacher complexity (RC). One could impose probability distribution on the loss function class with RC.
