Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Transformers.
to leverage the order of tokens, damages translation-invariance. To alleviate the limitations, LPU is proposed to extract local information using depth-wise convolution (MobileNetV1), which is defined as: 1.2. Lightweight Multi-Head Self-Attention (LMHSA) In original self-attention module, the