Thanks for the code!
Since CUDA operations are asynchronous, you should synchronize the code before starting and stopping the timer via torch.cuda.synchronize().
Also, generally you should add some warmup iterations to get stable results.
Especially, if you are using torch.backends.cudnn.benchmark = True, as the first iterations will time different kernel implementations and will chose the fastest ones for your workload.
Adding regularization like Dropout, a more aggressive data augmentation, early stopping etc. could help countering overfitting.