They have the same distribution and processing is identical for both sets.
When I disable normalization inside the model and perform it using torchvision library things get back to normal. I suppose the weird performance drop is purely from normalization inside the model.