One more thing to notice is that you’re passing a weight_decay value of 0.0 for both parameter groups in your optimizer. This is harmless (and kinda pointless) when using the default value (which is also 0.0), but otherwise it can certainly lead to different results.