I might be mistaken, but I thought weight normalization would be a reparametrization of the weight parameter, which should yield faster convergence. The output should be the same as without weight norm.
I just skimmed though the original paper 6 again, but please correct me, if I’m wrong.