I apologize if I’ve missed something obvious here - this question relates to issues I am having timing mixed precision vs float32 computation.
I have two servers - one with Pytorch 1.5 and Cuda 10.1, and the other with Pytorch 1.6 and Cuda 11.0. As far as I know there are no Pytorch CUDA 11.0 binaries, so that pytorch was compiled with 10.1. Both have 2080 Ti RTX GPU cards.
In both servers, I time pure fp32 computation as being significantly faster than mixed precision and I can’t work out why.
I am aware that for the Pytorch1.6 server the mismatch between CUDA versions is not ideal, but I’m still not sure why there should be an issue on the Pytorch 1.5 server.
My times are as follows:
Pytorch 1.5:

Pytorch 1.6

I’ve attached gists for the two scripts that I’m using to compute times below.

  
      gist.github.com
  
  
    https://gist.github.com/Ushk/96be97cd595ee63ae5cd6cebe415e642 12
gistfile1.txt
import random
import os
import numpy as np
import time
import torch
import torch.nn as nn
from apex import amp

def set_seed(seed: int):
    """Set all seeds to make results reproducible (deterministic mode).
This file has been truncated. show original




  
  
    
    
  
  



  
      gist.github.com
  
  
    https://gist.github.com/Ushk/8831348c071ddea6a90c7dddad124d14 11
gistfile1.txt
import os
import time
import torch
import torch.nn as nn
import numpy as np
import random
from apex import amp
from torch.cuda.amp import autocast

def set_seed(seed: int):
This file has been truncated. show original




  
  
    
    
  
  


Torch versions:
image719×180 20.2 KB
image730×178 20.3 KB
Output of nvcc --version


Nvidia-smi:
image805×354 35.8 KB
image809×394 36.4 KB
Based on this comment/thread:

  
      github.com/pytorch/fairseq
  
  
    
  
	  
  

  
    
      Huge Training time for FP16 on Turing/2080 GPU 9
    

    
      
        opened 
        
          
        
        Jan 22, 2019
      
      

        
          closed 
        
          
        
        Feb 19, 2019
      
        

      
        
          
          gvskalyan
        
      
    
  



  I am training en-fr on GeForce RTX 2*2080 ti with nvidia driver version 410.79 (although the latest is 410.93) with CUDA...





  
  
    
    
  
  


I would expect there to be a speedup on a 2080 Ti  - is this correct?