I see!
One suggestion for debugging, can you try using the same batch of data multiple times, and see if the error is decreasing or no? With the same batch of data, the weights should also be fixed. So, just pass the same batch a few times, and compute the loss, and then do bacward loss.backward() and update the model via optim.step(). Then, you can look at the loss and see if the loss is decreasing or no.