torch.utils.benchmark would probably give you the most accurate estimate, as it will add warmup iterations and synchronize for you.
Different performance on different devices might be expected especially if you are using libs to accelerate the workload, e.g. cuDNN.