How did you measure the performance for your CPU and GPU runs?
Did you properly synchronize the code for the GPU run before starting and stopping the timer?
If so, your scripts might either face other unrelated bottlenecks, which might not give you a proper timing, and should thus be removed (e.g. remove the data loading and use random input tensors).
Also, torch.backends.cudnn.benchmark = True will profile the kernels in the first iteration for each new input shape and will select the fastest one, which might yield a speedup.
If all this was already done, then your model might suffer from a lot of kernels with tiny workloads, so that your overall GPU runtime might see the kernel launch overheads.