From your code it’s quite confusing as you don’t actually pass the weight_decay parameter to the optimizer in case 2. Is that just a typo? Or do you actually always have weight_decay to 0 and this has no link with weight decay?