For this problem BCELossWithLogits is a better choice than L1 loss.
Have you shuffled the data properly? Another reason would be, that in one-minibatch the dominating background say is of color (12, 12, 12) and in other minibatch background is of color(200, 200, 200). Expanding on this idea, your data in different dataloader steps is most probably biased towards some class.