Could you remove the inplace modification of the loss tensor and use:
loss = loss + loss_fn(...)

instead?
Based on the posted code snippet it seems to be the first place I would start to debug the issue.