Thanks for the correction. I missed the average in the calculation.
You could set torch.backends.cudnn.benchmark = True at the beginning of your script to use the cudnn heuristics in order to find the fastest kernels for your current workload.
Additionally, you could check out automatic-mixed precision 3 (available in master and the nightly binaries), especially if you are using a newer GPU with TensorCores (which unfortunately shouldn’t help much with the GTX 750).