I have a similar experience with poor PyTorch performance compared to TensorFlow. Here’s a simple script that you can test for yourself.
Tensorflow runs at least 4x faster than PyTorch on this notebook using GPU, TPU, CPU at Google’s Colab.

  
      
      colab.research.google.com
  
  
    

Google Colaboratory 4



  
  
    
    
  
  


I used the following script to time the training cells:
import time
# ----------------------------
start_time = time.time()
# ----------------------------
_ = modelTF.fit(x_trainTF, y_trainTF, epochs=epochs, batch_size=batch_size, verbose = 0)
print("--- %s seconds ---" % (time.time() - start_time))

# ----------------------------
start_time = time.time()
# ----------------------------
for e in range(epochs):
    for images, labels in xy_trainPT_loader:
        optimizer.zero_grad()
        images = images.view(images.shape[0], -1) 
        loss = criterion(modelPT(images), labels)        
        loss.backward()
        optimizer.step()
        
print("--- %s seconds ---" % (time.time() - start_time))