Thank you for your timely response!
I found the origin of those magic numbers in torchvision.models doc 7, which, as you said, should be statistics of ImageNet.

All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using  mean = [0.485, 0.456, 0.406]  and  std = [0.229, 0.224, 0.225] …

However, it also said as long as users would like to use those pretrained models, they must normalize their images in this way. I think this makes sense since those models were trained on data scaled according to those specified mean and std.