Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Transformers.
Attention on Higher Resolution Attention on Higher Resolution To perform MHSA at the earlier stages of the network, all Query, Key, and Value are downsampled to a fixed spatial resolution (1/32) and the outputs are interpolated from the attention back to the original resolution to feed into the