All weight normalization variants restrict magnitudes, so there should be some layer that can rescale features freely.



 michaelklachko:

If a channel is “dead” with weights very close to zero, do we want to rescale it to be as strong as other, useful channels? Simply scaling down all channels by the same values preserves the relative channel importance information. I wonder if batch normalization does some form of channel equalization, but I believe it still allows some channels to go “dead”.


It is an intermediate representation, so there is no value in “dead” channels. On the contrary, you get problems with vanishing gradients in multi-layer pipelines. Indeed, batch normalization helps to avoid “dead” channels (affine transformation it does is kinda auxiliary, I think).



 michaelklachko:

Though I still don’t quite understand what happens to gradients, the computational graph does not make much sense to me.


You have a special cell in a weight matrix (located at argmax()) that all channels use to try to adjust outputs. Row that contains this cell thus produces a feature with different gradient magnitude and/or variance. So, SGD suffers, while Adam adjusts to mitigate this.