In the default setup your loss will be the average or sum of your batch.
You can set reduce=False to get the same batch size for your loss:
batch_size = 10
criterion = nn.MSELoss(reduce=False)
x = torch.randn(batch_size, 2, requires_grad=True)
y = torch.randn(batch_size, 2)
loss = criterion(x, y)