I suppose that something it is going to change with the new Tensorflow runtime:

  

      github.com/tensorflow/tensorflow
  

  
    
  
	  
  

  
    
      How to create new cuda stream in custom op 5
    

    
      
        opened 
        
          
        
        Feb 11, 2020
      
      

        
          closed 
        
          
        
        Apr 3, 2020
      
        

      
        
          
          chaaland
        
      
    

    
        
          stat:awaiting tensorflower
        
        
          type:support
        
        
          comp:gpu
        
    
  


  
    I've written a custom op for the GPU that takes X with shape ```(n,p)``` and Y w…ith shape ```(m,p)``` and returns Z with shape ```(n,m)```. 

In the backward pass, dX and dY are independent computations so I'd like to have launch two separate CUDA kernels and have them run in separate streams. I know the cuDNN RNN is capable of doing kernels in parallel but I cannot find the source code for how to do this. It seems the only usable stream is provided by the OpKernelContext. 

Matrix multiplication must have a similar parallelism when doing the backward pass but I have not found anywhere in the source making use of multiple streams. This is essentially a reopening of [this issue](https://github.com/tensorflow/tensorflow/issues/6675) as there doesn't appear to a public answer