How did you measure the timings and did you synchronize both codes?
I’m not familiar enough with TensorFlow and don’t know how the code can be synchronized, but in PyTorch you should call torch.cuda.synchronize() before starting and stopping the timer. Otherwise your measurements are invalid.