Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Transformers.
2: Consistent feature dimension is important for the choice of token mixer. MHSA is not necessarily a speed bottleneck. Observation 3: CONV-BN is more latency-favorable than LN (GN)-Linear and the accuracy drawback is generally acceptable. Observation 4: The latency of nonlinearity is hardware and