You would usually calculate the stats from the training dataset, since the validation and test sets are “unknown” and using their stats could be considered as a data leak.


I would claim it depends on the overall use case, i.e. are you able to use different transformations for each modality during testing (is the modality a separate input you’ll get)? If so, you could calculate and apply different stats to each input type. If you are only getting an image and don’t know the modality, I would claim you should try to use the mixed stats. However, this approach might not standardize the data properly (zero mean and unit variance) if the input modalities have different stats.