Your loss might get a zero value, which would create an Inf or NaN gradient:
x = torch.randn(1, 1, requires_grad=True)
y = torch.zeros(1, 1)
criterion = nn.MSELoss()

loss = torch.sqrt(criterion(x * 0, y))
loss.backward()
print(x.grad)
> tensor([[nan]])

You could add a small eps value to the torch.sqrt calculation to avoid it.