Is Glorot/He-style variance-preserving *regularization* a known thing? In the context of deep learning, layer weight initialization techniques of the Glorot/He family strive to select initial random weight values in such a way that the variance of each layer's output is equal to the variance of its input. This helps avoid vanishing/exploding gradient issues and improves training stability.
It seems likely to me that this is a desirable property to maintain not only initially, but also during training, as a regularization constraint via an explicit loss term nudging weight distribution to the optimal distribution mandated by Glorot/He.
Has this been studied?

"Self-Normalizing Neural Networks" by Günter Klambauer Thomas Unterthiner Andreas Mayr & Sepp Hochreiter proposes a neural network that converges to activations with zero mean and unit variance.

Deep Learning has revolutionized vision via convolutional neural networks (CNNs)
and natural language processing via recurrent neural networks (RNNs). However,
success stories of Deep Learning with standard feed-forward neural networks
(FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot
exploit many levels of abstract representations. We introduce self-normalizing
neural networks (SNNs) to enable high-level abstract representations. While
batch normalization requires explicit normalization, neuron activations of SNNs
automatically converge towards zero mean and unit variance. The activation
function of SNNs are “scaled exponential linear units” (SELUs), which induce
self-normalizing properties. Using the Banach fixed-point theorem, we prove that
activations close to zero mean and unit variance that are propagated through many
network layers will converge towards zero mean and unit variance — even under
the presence of noise and perturbations. This convergence property of SNNs allows
to (1) train deep networks with many layers, (2) employ strong regularization
schemes, and (3) to make learning highly robust. Furthermore, for activations
not close to unit variance, we prove an upper and lower bound on the variance,
thus, vanishing and exploding gradients are impossible. We compared SNNs on
(a) 121 tasks from the UCI machine learning repository, on (b) drug discovery
benchmarks, and on (c) astronomy tasks with standard FNNs, and other machine
learning methods such as random forests and support vector machines. For FNNs
we considered (i) ReLU networks without normalization, (ii) batch normalization,
(iii) layer normalization, (iv) weight normalization, (v) highway networks, and (vi)
residual networks. SNNs significantly outperformed all competing FNN methods
at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and
set a new record at an astronomy data set. The winning SNN architectures are often
very deep.

