Test (Nature images:  Intel Image Classification | Kaggle 3)
Experiment 1.1: ImageNet Custom Stats vs None
ResNet-18:

resnet-18942×617 48.4 KB

ResNet-34:

Screenshot from 2021-04-01 09-13-46654×628 50.1 KB

You can see the general trend being that the normalised experiments are slightly above the experiment without normalisation most of the epochs.
Conclusion 1: Using custom ImageNet stats for the pytorch pretrained models provides better results than without normalisation as it is trained on ImageNet.
Experiment 1.2: ImageNet Custom Stats vs Own Custom Stats
ResNet-18:

Screenshot from 2021-04-01 10-10-311269×639 94.3 KB

ResNet-34 (Unstable, need to go more epochs to smooth out the accuracy):

Screenshot from 2021-04-01 11-34-091263×655 117 KB

Conclusion 2: Using your own dataset stats gives better result.
For further experiments into other domains like medical images and stock technical charts photos, I think it is unnecessary as it is obvious to me that your own dataset custom stats will be superior.
Personal Opinion: Just always find your own dataset mean and standard deviation, normalise accordingly. Keep this as a common practice.  It only takes roughly 5 minutes to write the script and run it.