Replacing softmax with ReLU in Vision Transformers
Previous research observed accuracy degradation when replacing the attention
softmax with a point-wise activation such as ReLU. In the context of vision
transformers, we find that this degradation is mitigated when dividing by
sequence length. Our experiments training small to large vision transformers on
ImageNet-21k indicate that ReLU-attention can approach or match the performance
of softmax-attention in terms of scaling behavior as a function of compute.
6507b98ac53e1a7f17d7f76c
Mitchell Wortsman
admin_assigned
2023-09-18T09:59:30.631Z
638e42f96251c8bd7ab84089
/avatars/cb7a7f902846bd9710e2c48a50ce3527.svg
Mitchell Wortsman
user
mitchellw
6507b98ac53e1a7f17d7f76d
Jaehoon Lee
6507b98ac53e1a7f17d7f76e
Justin Gilmer
6507b98ac53e1a7f17d7f76f
Simon Kornblith
admin_assigned
2023-09-18T10:00:16.940Z
6443ffc2af034cdfd69c5c00
/avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
Simon Kornblith
user
simonsterk
Replacing softmax with ReLU in Vision Transformers
Previous research observed accuracy degradation when replacing the attention
softmax with a point-wise activation such as ReLU. In the context of vision
transformers, we find that this degradation is mitigated when dividing by
sequence length. Our experiments training small to large vision transformers on
ImageNet-21k indicate that ReLU-attention can approach or match the performance
of softmax-attention in terms of scaling behavior as a function of compute.
Replacing the attention softmax with ReLU and dividing by sequence length in vision transformers can maintain or improve performance compared to using softmax alone.
attention softmax ReLU-attention sequence length vision transformers