oh there is weight_decay option… weight_decay is like l2 regularization so it will change your weight if it is nonzero. By default in optim.Adam it is zero though. It could be that the script has nonzero default value. Could you try setting that to 0?