I see. So the forward is supposed to use normalized weights? You can still do it as an optimizer though. Just store the unnormalized weights in optimizer state dict and in each step update the model parameters to normalized weights.