Thank you for the reply, @ptrblck !
Indeed, this happens on test images, which can be quite different from the training dataset.
I will try to include some augmentation on the train data, so that it resembles the test data better.