Thank You, this solved my issue. I forgot to zero out gradients after each minbach. Now works fine.
# Optimizer needs the gradients of this minibatch only, so zero out prev grads.
optimizer.zero_grad()
loss.backward() # Calculates derivatives with autograd
optimizer.step() # Update weights