Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Transformers.
self-attention module is: To mitigate the computation overhead, a k × k depth-wise convolution with stride k (MobileNetV1) is used to reduce the spatial size of K and V before the attention operation. Also, a relative position bias B (Similar to Shaw NAACL’18) is added to each self-attention