I also ran into this issue. And found the solution here,

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      RuntimeError: CUDA error: an illegal memory access was encountered with channels_last 7
    

    
      
        opened 
        
          
        
        Apr 28, 2020
      
      


      
        
          
          tstandley
        
      
    
  



  I get an illegal memory access when trying to train mnasnet (any version) with apex (O1) and channels_last
To Reproduce
Steps to reproduce...



    high priority
    module: binaries
    module: cuda
    module: cudnn
    topic: dependency bug
    triaged


  
  
    
    
  
  


Apparently the latest cudnn v8 has it fixed.