This is really a CUDA issue, see thediscussion in

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Massive initial memory overhead GPU 102
    

    
      
        opened 
        
          
        
        Oct 19, 2018
      
      

        
          closed 
        
          
        
        Apr 15, 2019
      
        

      
        
          
          davidmascharka
        
      
    
  



  🐛 Bug
There is a huge RAM overhead for using the GPU even for processing small tensors.
Here's a standalone script:
# test.py
import torch
import...





  
  
    
    
  
  


One option might be to remove unneeded kernels. This is done for mobile already and discussed here:
  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [Feature Request] reduce CUDA runtime size by selectively compiling PyTorch GPU kernels 81
    

    
      
        opened 
        
          
        
        Jan 6, 2020
      
      


      
        
          
          xxradon
        
      
    
  



  🚀 Feature
pytorch massive initial memory overhead GPU,we need a way to control the memory and gpu memory during pytorch to cuda()...



    module: binaries
    module: build
    module: cuda
    triaged


  
  
    
    
  
  


It’s not too hard to leverage the mobile work if you know how to do it.