Hello,
When using something like the following
            for param in net.parameters():
                loss += weight_decay * 0.5 * torch.pow(param, 2).sum()

instead of the weight_decay option of the optimizer, depending on the weight_decay parameter,
i notice different updates after some epochs compared to using the latter method.
These differences ultimaltely lead to a higher variance when evaluating a model over multiple random seeds.
Where do these differences come from and how can i get rid of them?
Thanks in advance!