title: Replacing softmax with ReLU in Vision Transformers
summary: Previous research observed accuracy degradation when replacing the attention
  softmax with a point-wise activation such as ReLU. In the context of vision
  transformers, we find that this degradation is mitigated when dividing by
  sequence length. Our experiments training small to large vision transformers on
  ImageNet-21k indicate that ReLU-attention can approach or match the performance
  of softmax-attention in terms of scaling behavior as a function of compute.
paper_authors._id: 6507b98ac53e1a7f17d7f76c
paper_authors.name: Mitchell Wortsman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T09:59:30.631Z
paper_authors._id: 638e42f96251c8bd7ab84089
paper_authors.avatarUrl: /avatars/cb7a7f902846bd9710e2c48a50ce3527.svg
paper_authors.fullname: Mitchell Wortsman
paper_authors.type: user
paper_authors.user: mitchellw
paper_authors._id: 6507b98ac53e1a7f17d7f76d
paper_authors.name: Jaehoon Lee
paper_authors._id: 6507b98ac53e1a7f17d7f76e
paper_authors.name: Justin Gilmer
paper_authors._id: 6507b98ac53e1a7f17d7f76f
paper_authors.name: Simon Kornblith
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:00:16.940Z
paper_authors._id: 6443ffc2af034cdfd69c5c00
paper_authors.avatarUrl: /avatars/0c12cf9baaf3b24a02ffb42238cdc6df.svg
paper_authors.fullname: Simon Kornblith
paper_authors.type: user
paper_authors.user: simonsterk
paper_title: Replacing softmax with ReLU in Vision Transformers
paper_summary: Previous research observed accuracy degradation when replacing the attention
  softmax with a point-wise activation such as ReLU. In the context of vision
  transformers, we find that this degradation is mitigated when dividing by
  sequence length. Our experiments training small to large vision transformers on
  ImageNet-21k indicate that ReLU-attention can approach or match the performance
  of softmax-attention in terms of scaling behavior as a function of compute.
paper_ai_summary: Replacing the attention softmax with ReLU and dividing by sequence length in vision transformers can maintain or improve performance compared to using softmax alone.
paper_ai_keywords: attention softmax ReLU-attention sequence length vision transformers