This reminds me of Normalized Direction-preserving Adam 3. Regarding your approach, you should probably use vector maximums instead of a single scalar divisor.
Also, nn.utils.weight_norm 19 may be relevant.