So I compared the result to the same network, implemented in Pyton with the same seed and all. Turns out the resulting weights are not the same between this naive C++ implementation and pytorch. I guess there’s more happening inside pytorch than I thought, especially in the backwards pass.
Has somebody figured out how to implement weight_norm? Or is it just best to wait for official support in upcomming Libtorch releases?