There is an active issue on this, to break multi head attention into parts, as it is a bit too long.
I think they will update it soon.

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [FYI] MultiheadAttention / Transformer 207
    

    
      
        opened 
        
          
        
        Jan 24, 2020
      
      


      
        
          
          zhangguanheng66
        
      
    
  



  This issue is created to track the progress to refine nn.MultiheadAttention and nn.Transformer.
Since the release of both modules in PyTorch v1.2.0,...



    module: nn
    proposal accepted
    triaged