Confining the function approximation of NN I was recently studying about LMSE estimates and came across an idea. Is there a technique to reduce the hypothesis space of a Neural Network by introducing regularisation techniques of some sort similar to how we reduce function spaces in estimates? One method that immediately comes to my mind is to change the activation functions and manipulate them somehow but it isnt so obvious to me how it might work.
EDIT: I think my question has been misunderstood. Iterating over learnable activation functions is probably one solution to what I am thinking of. Specifically, I think my question is " Similar to how the LMSE loss searches over an affine subspace of estimators, is there a similar logic in NNs to make them search for an approximation of any arbitrary function using a certain "type" of estimators i.e say only piecewise constant approximations as done by the Haar scaling function in wavelets.

Regularization penalizes regions in the hypothesis space more than others. So, to use regularization on activation functions, those activation functions would have to become part of the hypothesis space, which they are not in normal NNs. You would have to make your activation functions depend on parameters, which would then be part of the optimization. Now, this results in a new architecture, which depending on how this parametric dependence looks like, might not really be called an NN anymore. Many questions come up, like, what would be, exactly, the parametric dependency of the activation functions, will those new architectures still be as well trainable as ordinary NNs, nobody knows whether backpropagation would still be really appropriate, and so on.
There are already steps in this direction of using learnable activation functions. E.g. maxout networks can be understood as having learnable activation functions. See also here and here.
