So I did that and it turns out that MeanSquaredError metric has a different output than the Loss = torch.nn.MSELoss() even on the first batch.
From running:
        loss.backward()
        optimizer.step()

        mse_metric.reset()
        loss_metric.reset()

        mse_metric.update((y_pred, y))
        loss_metric.update((y_pred, y))

        print(mse_metric.compute())
        print(loss_metric.compute())

I got :
8.566025390625
0.05948628857731819