I have moved my model from V100 to A100 and instead of seeing an increase in speed these has been a significant slowdown from 14.2 it/sec to 10.06 it/sec.
cuda version 11.3
Pytorch version 1.9.0+cu111
I have been specifically using the code from GitHub repository (NLSPN)
  
      

      GitHub
  

  
    

GitHub - zzangjinsun/NLSPN_ECCV20: Park et al., Non-Local Spatial Propagation... 2

  Park et al., Non-Local Spatial Propagation Network for Depth Completion, ECCV, 2020 - GitHub - zzangjinsun/NLSPN_ECCV20: Park et al., Non-Local Spatial Propagation Network for Depth Completion, ECC...


  

  
    
    
  

  


There is a apex dependency in the repository which I thought to be the issue, but removing it and using a single GPU training also suffers from this issue.