Hello @Xinwei_he,
A small doubt here, Are you adding spectral Norm of the weight as an extra regularizer ? Or Are you embedding it with SGD algorithm itself?
Actually I am trying to see the effect of Spectral norm of weight as an regularizer (like l2 loss). Would your approach be useful in my case?( Since the function calculating spectral norm wouldn’t be differentiable), And when I am adding to the total loss function, I should add something is differentiable.
Thanks!