If there are parameters you don’t want to optimize you should set their requires_grad flag to False. .eval() only changes the behaviour of modules like dropout or batch norm and should not be enabled during training.