I just checked! Only the last batch has different shapes:
image shape itr 81:  torch.Size([1, 3, 300, 300])
target shape itr 81:  torch.Size([1])

What should I do?  Should I force the training not to use that specific batch? why does this happen?