Sorry for late.
As far as I know:

If you want to use a pre-trained network to use on your own dataset, for training phase, you should use mean, std on new dataset, and if you only want to get the performance of the pre-trained network on you own dataset, I am not sure substract means and std of ImageNet will work.  

when calculate mean and std, only on train dataset is enough, because valid dataset and test dataset is used to mesure your model performance. And calculate on original images.
I tried to calculate them on cpu, but it really cost so much and if the dataset is large, it will spend much time, but I think it is okay, because it only computed one time.
transforms.normalize option is after torchvision.transforms.functional.to_tensor if the data convert to tensor, its value is [0,1].