The second multi-head attention is contextual attention, which leverage the output from encoder and decoder self-attention. I don’t quite understand attn_mask and key_pad_mask in the contextual attention, and wondering whether could you explain those two masks? Thank you!

If “the contextual attention” means the encoder-decoder attention of the Transformer, they (attn_mask and key_pad_mask) are memory_mask and memory_key_padding_mask.