Here is a simple example you could check and play around with to see the effect of the previous description:
data = torch.randn(20, 1)
target = torch.randn(20, 1)
criterion = nn.MSELoss()

# compute loss of entire dataset
loss_ref = criterion(data, target)

# compute loss using mini batches
dataset = torch.utils.data.TensorDataset(data, target)
loader = torch.utils.data.DataLoader(dataset, batch_size=8)
# loader will return 3 batches with [8, 8, 4] samples

# average by number of batches
loss_avg1 = 0.
for data, target in loader:
    loss = criterion(data, target)
    loss_avg1 += loss.item()
loss_avg1 = loss_avg1 / len(loader)

# average by number of samples
loss_avg2 = 0.
for data, target in loader:
    loss = criterion(data, target)
    loss_avg2 += loss.item() * data.size(0)
loss_avg2 = loss_avg2 / len(loader.dataset)

# compare
print('reference {}, avg1 {}, avg2 {}'.format(
    loss_ref, loss_avg1, loss_avg2))

You could write down the formula used to create the final losses for all use cases and check where the error is coming from. To do so, note that the criterion calculates the mean loss by default.