Do your training and validation images have the same distributions or are you processing them differently?
Are you seeing the bad results also after calling eval() and using your training images?
If so, this problem might be unrelated to the processing inside the model, but might come from a small batch size and thus skewed running estimates in the batch norm layers.