I am learning attention mechanism through reading official tutorial ‘Translation with a Sequence to Sequence Network and Attention’
but I think there is something wrong about implementation of attention decoder:

image.png776×654 69.8 KB

we can find that attention weights are calculated by code below：
attn_weights = F.softmax(self.attn(torch.cat((embedded[0], hidden[0]), 1)))
however，the dimention of hidden is (num_layers * num_directions, batch, hidden_size)
while the dimention of embedded is  (batch, seq_len, embedding_dim) or (seq_len, batch, embedding_dim)
Obviously seq_len is usually not equal with  num_layers * num_directions ， so it is  reasonable to concatenate hidden and embedded？
I am a beginner of NLP and hope that someone who understand the tutorial can explain the reason why they concatenate hidden and embedded, Thank you!