I would recommend to run a quick profiling code and just get the runtimes for your current workload and setup. 
Also, if you are using the GPU, don’t forget to synchronize the code via torch.cuda.synchronize().