Is Glorot/He-style variance-preserving *regularization* a known thing? In the context of deep learning, layer weight initialization techniques of the Glorot/He family strive to select initial random weight values in such a way that the variance of each layer's output is equal to the variance of its input. This helps avoid vanishing/exploding gradient issues and improves training stability.
It seems likely to me that this is a desirable property to maintain not only initially, but also during training, as a regularization constraint via an explicit loss term nudging weight distribution to the optimal distribution mandated by Glorot/He.
Has this been studied?

There are ways to preserve activation variance with an explicit regularization term.  (As opposed to implicit regularization via batchnorm, self-normalizing networks, etc.)  For example, the orthogonality regularizer
$$
\hat{\ell}(\theta) = \sum_k || {\theta_k}^T \theta_k - c_k I ||^2
$$
will do it, given square weight matrices $\theta_k$.  (Possibly for rectangular $\theta_k$ as well, but I can't recall or verify at the moment.)
Above, $\theta$ are network weights, with $\theta_k$ the weight matrix for each layer $k$.  $I$ is the identity matrix, and $c_k$ is a scalar gain factor depending on the chosen nonlinearity and on data distribution properties.  See details below.

Explanation
To state the goal a bit more formally, for layer $k$ in a neural net:

*

*Let $\theta_k$ be layer parameters, so that the input $x$ and output $y$ of the layer are written $y = f_{\theta_k}(x)$

*Let $X_k$ and $Y_k$ be random variables representing layer $k$ input and output, so that we can talk about their distributions.  i.e. $Y_k = f_{\theta_k}(X_k)$
Given a training sample $z$ and network weights $\theta = \{\theta_1, \theta_2, \dots\}$, we seek a regularization function $\hat{\ell}(\cdot)$ so that the overall objective function
$$
\mathcal{L}(z; \theta) = \underbrace{\ell(z; \theta)}_{\text{loss}} + \lambda \underbrace{\hat{\ell}(\theta)}_{\text{regularizer}}
$$
induces
$$
\text{Var}(Y_k) \approx \text{Var}(X_k)
$$
for each layer $k$, during and after optimization.

To start simply, assume that $\theta_k$ is a $d \times d$ square matrix and $f_{\theta_k}: \mathbb{R}^d \to \mathbb{R}^d$ is linear, so that the layer is just a matrix multiplication:
$$
\begin{align}
Y_k &= f_{\theta_k}(X_k) \\
&= \theta_kX_k
\end{align}
$$
In this case, enforcing orthogonality of $\theta_k$ is a simple way to preserve variance, regardless of the properties of $X_k$.  Enforcing or encouraging orthogonality during gradient descent is a well-studied problem.  Since orthogonality can be defined as ${\theta_k}^T \theta_k = I$, a simple regularizer just minimizes the distance between both sides of the equation for each layer:
$$
\hat{\ell}(\theta) = \sum_k || {\theta_k}^T \theta_k - I ||^2
$$
Hence, each weight matrix $\theta_k$ is pushed towards orthogonality, with the proximity to orthogonality controlled by regularization weight $\lambda$ in the objective function $\mathcal{L} = \ell + \lambda \hat{\ell}$
Note, this preserves variance for linear $\mathbb{R}^d \to \mathbb{R}^d$ layers only.  As we reintroduce practicality to our layers (i.e. nonlinear activation functions, non-square weight matrices, or even convolutions) then more care is necessary to preserve variance.  For example, if we assume $X_k \sim N(0, \sigma^2)$ and $f_{\theta_k}$ uses ReLU activation, then $\text{Var}(Y_k) = c \sigma^2$ for some constant $c$.  So we should replace the identity matrix $I$ with $\frac{1}{\sqrt{c}} I$ in the regularizer $\hat{\ell}(\cdot)$.  (I am too lazy to derive this particular $c$ right now, but empirically, $c \approx 0.34$.)  .
Different nonlinearities will have different gain factors.  For example, I think OPLU conveniently has a gain factor of 1.  These gain factors are discussed in a few different places, including the orthogonal initialization and descent literature, and the initialization literature in general.
Instead of soft-constraint by regularization, we can also hard-constrain to orthogonality by various methods.  For instance, this paper uses weight orthogonality to address vanishing/exploding gradients in recurrent networks.  But since you asked only about regularizers, I won't detail anything about that.  And typically, if one is interested in orthogonality-preserving optimization, then one is also interested in orthogonal weight initialization.
(The papers linked here are not comprehensive, and probably a bit out of date.  If anyone knows of relevant surveys to link instead, do share in the comments.)
