So I don’t know how but I don’t seem to be getting this problem anymore. Normally though if I change the batch size I can see the allocated memory decrease but in this situation it was always maxed. I have encountered this behavior in TensorFlow but never before in Pytorch. Sometimes (but rarely) this would cause an OOM randomly during an epoch (was using static shapes).
Interestingly I also upgraded my Pytorch version which resulted in VRAM halved and an epoch speed up by ~33%, which is pretty insane…
Edit: The performance improvements seems to be specific to the 30-series GPU (was just slow to notice this):
  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Convolution operations are extremely slow on RTX 30 series GPU
    

    
      
        opened 
        
          
        
        Oct 29, 2020
      
      

        
          closed 
        
          
        
        Oct 29, 2020
      
        

      
        
          
          P3n9W31
        
      
    
  



  🐛 Bug
I'm using PyTorch on an RTX 3090 GPU, and trying to train different types of networks (e.g., CNN, RNN, Transformer),...