Your benchmarking code isn’t accurate. CUDA kernels are launched asynchronously, but you’re measuring the time using time.time() even though the forward pass won’t have completed at that point.
To my knowledge, both PyTorch and TensorFlow use same implementation (cuDNN) for convolutions. The speed of the convolution implementation is going to dominate if you have sufficient batch size (which you do). The other differences you see are probably differences in measurement (see point above) and how you’re passing in data.
There’s already a bunch of accurate benchmarks of CNNs:

  
      
      GitHub
  
  
    

soumith/convnet-benchmarks 967

convnet-benchmarks - Easy benchmarking of all publicly accessible implementations of convnets


  
  
    
    
  
  



  
      
      GitHub
  
  
    

jcjohnson/cnn-benchmarks 731

cnn-benchmarks - Benchmarks for popular CNN models