Hi ptrblck,
I also need to do weight initialization in current project, but I need to do it in every forward pass.
So the torch.no_grad() method is not suit for me.
I found the solution in here 76.
self.pred.weight = torch.nn.Parameter(self.pred.weight / torch.norm(self.pred.weight, dim=1, keepdim=True))

I wanna know those cast operation(cast Parameter to Tensor) will affect the gradient flow or not ?
Or there is a better way to do weight normalization in forward pass ?
Thank you so much.