IMO, this question is barely related to pytorch since it’s a transformer architecture and MultiHeadAttention is an inner element for the Encoder and Decoder layers.
Anyway, here’s an explanation from architecture standpoint:
  

      stats.stackexchange.com
  

  
      
    
  


  What exactly are keys, queries, and values in attention mechanisms?



  neural-networks, natural-language, attention, machine-translation



  asked by
  
  
    Sean
  
  on 09:00AM - 13 Aug 19 UTC