Thanks for replying.
I cleaned up my environment, reinstalled and verified cudnn and reinstalled torch using pip. The code snippet above still fails for me, as is the following code:
import torch
from torch.backends import cudnn
torch.backends.cudnn.benchmark = True
out = torch.randn([2, 4, 3, 360, 640], dtype=torch.float, device=‘cuda’, requires_grad=True)
net = torch.nn.Conv3d(4, 1, kernel_size=[3, 3, 3], padding=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]).cuda()
out = net(out)
out.sum().backward()
Setting torch.backends.cudnn.benchmark = False succeeds. Additionally the following code succeeds:
import torch
from torch.backends import cudnn
torch.backends.cudnn.benchmark = True
out = torch.randn([2, 4, 3, 360, 640], dtype=torch.float, device=‘cuda’, requires_grad=True)
out.sum().backward()
I’ve seen similar issues reported in this forum and in github:
  
    
    
    CUDNN_STATUS_INTERNAL_ERROR for large image sizes on RTX GPU 
  
  
    torch.backends.cudnn.enabled = True uses cudnn, while ... = False disables it and falls back to the native PyTorch implementations. 
cudnn.benchmark = True could fail to find a fast kernel, but should fall back to a (slower but) working algorithm, which is not the case for your workload. 
Could you update PyTorch to the latest stable version (1.5) or the nightly binaries, as we’ve recently implemented another fallback mechanism, which should raise a proper error messge.
  


  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      cuDNN error: CUDNN_STATUS_INTERNAL_ERROR on cudnn.benchmark = True 2
    

    
      
        opened 
        
          
        
        Feb 7, 2019
      
      

        
          closed 
        
          
        
        May 5, 2019
      
        

      
        
          
          moderatelyfunctional
        
      
    
  



  Hi, I'm running PyTorch on an encoder/decoder architecture and am having a problem with cuDNN.
If I include
import torch.backends.cudnn as cudnn
cudnn.benchmark =...



    module: cudnn
    module: dependency bug
    triaged


  
  
    
    
  
  


  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      F.conv2d() causes RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR 
    

    
      
        opened 
        
          
        
        Oct 2, 2020
      
      


      
        
          
          timothybrooks
        
      
    
  



  🐛 Bug
I am receiving a RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR when calling F.conv2d().
Shapes and devices of inputs:
input, torch.Size([4, 256, 128, 128]), cuda:0...



    module: cudnn
    triaged


  
  
    
    
  
  


  
      github.com/microsoft/Bringing-Old-Photos-Back-to-Life
  
  
    
  
	  
  

  
    
      cuDNN error: CUDNN_STATUS_INTERNAL_ERROR 2
    

    
      
        opened 
        
          
        
        Nov 29, 2020
      
      

        
          closed 
        
          
        
        Dec 2, 2020
      
        

      
        
          
          wang7143
        
      
    
  



  Running Stage 1: Overall restoration
Now you are processing 5d6bfe49ge1b23ec32641&690.jpg
Skip 5d6bfe49ge1b23ec32641&690.jpg due to an error:
cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this...





  
  
    
    
  
  


  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR in 1.1.0 
    

    
      
        opened 
        
          
        
        Jun 21, 2019
      
      

        
          closed 
        
          
        
        Sep 24, 2019
      
        

      
        
          
          Moondra
        
      
    
  



  I am currently using Windows 7, Pytorch version 1.1.0,
It seems from what I have read this error should have been fixed
in...



    module: convolution
    module: cuda
    module: cudnn
    module: nn
    triaged


  
  
    
    
  
  


Frankly I’m not sure how to proceed with this and setting benchmark=False is not the solution I’d want.