Jean-Eric_Campagne:

Do you mean that 100k input images with data augmentation (flip/rotations) for resnet20 or higher resnetXXX is not enougth ?


It might still produce skewed running estimates, e.g. if you are dealing with 100 different distributions each creating 1000 samples.
E.g. if you are using 100k ImageNet samples and mix them with 100k CT scans, I would assume that the stats of the data will be somewhat inbetween, which would make the validation and test bad.