Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Transformers.
the Cross-Attention (CA) can be expressed as: Since only CLS is used in the query, the computation and memory complexity of generating the attention map (A) in cross-attention are linear rather than quadratic as in all-attention. Multiple heads are used, i.e. MCA: However, FFN is NOT used. LN is