@111179 Yeah I was detaching the tensors from gpu to cpu before the model starts learning. So in that case the optimizer and the learning rate does affect anything. What particularly your model is doing?