Well, here, you have a batch size of 4 and one of 2 and set up MSELoss to average over the batch items.
If you concatenate, you take MSELoss to average over all 6 items and it effectively computes the weighted average (4*8630+2*2850)/6 = 6703.
When using a typical setup with PyTorch’s DataLoader, all batches but the last (and there is the drop_last argument to just leave it out) have the same size, so the weights in the averaging would be all the same.