I’m sorry I didn’t mention in the code above.
But, this is exactly what I used. Even then there is no change in loss.
In train loop:
optimizer.zero_grad()
loss = model.training_step()
loss.backward()
optimizer.step()