Hi @albanD,



 albanD:

In practice, most of our kernels actually use the whole GPU and so kernels don’t run at the same time.


That’s not exactly how GPUs work. After a kernel issues all of its blocks, there is a “tail” of blocks that partially occupy the GPU. If there are just a few waves of blocks, then the tail can be a significant fraction of the total kernel execution time. Another kernel launched on a second stream could fill the device during the tail of the first kernel, increasing device utilization.
The issue for PyTorch would be that kernels that have a very short execution time will hit the limit of how fast you can launch kernels from the CPU (let alone from a python application).
CUDA Graphs were introduced to move kernel launch to the device side, reducing launch overhead for short running kernels.

  
      NVIDIA Developer Blog – 5 Sep 19
  
  
    

Getting Started with CUDA Graphs | NVIDIA Developer Blog 31

The performance of GPU architectures continue to increase with every new generation. Modern GPUs are so fast that, in many cases of interest, the time taken by each GPU operation (e.g.


  
  
    
    
  
  


Apparently there is an effort underway to use CUDA Graphs in PyTorch,

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Static Graphs using CUDA 10 Graphs API 17
    

    
      
        opened 
        
          
        
        Dec 30, 2018
      
      


      
        
          
          fps7806
        
      
    
  



  🚀 Feature
CUDA 10 released a new feature called CUDA Graphs which allows you to build static graphs that can minimizes the...





  
  
    
    
  
  


… but it seems to have hit a snag:

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Large portions of PyTorch test suite don't work properly on non-default stream 10
    

    
      
        opened 
        
          
        
        Jun 10, 2019
      
      


      
        
          
          ezyang
        
      
    
  



  In @fbhuba PR #21474 we switch the stream before running tests. But in this PR, a lot of tests failed and...



    high priority
    large
    module: cuda
    module: tests
    triaged


  
  
    
    
  
  


Anyway, as these issues are very interesting to me, I would like to learn more about what the roadmap looks like for this kind of functionality in PyTorch.
[Edit: fixed link to #15623]