Transformers, Deep Learning, Machine Learning, Editors Pick, Data Science.
architectures is that the decoder uses encoder-decoder attention, which uses both the outputs of the encoder (as K and V) and the inputs of the decoder block (as Q) during its attention calculation. This contrasts with self-attention, where the same input embedding matrix is used for all inputs.