Could you check, if trans creates the same output as your manual normalization?
If that’s the case, how reproducible is this issue? E.g. if you run the manual approach with 10 different seeds vs. the ToTensor approach, is the second one always worse than the first use case?