title: EquiformerV2: Improved Equivariant Transformer for Scaling to
    Higher-Degree Representations
summary: Equivariant Transformers such as Equiformer have demonstrated the efficacy of
  applying Transformers to the domain of 3D atomistic systems. However, they are
  still limited to small degrees of equivariant representations due to their
  computational complexity. In this paper, we investigate whether these
  architectures can scale well to higher degrees. Starting from Equiformer, we
  first replace SO(3) convolutions with eSCN convolutions to efficiently
  incorporate higher-degree tensors. Then, to better leverage the power of higher
  degrees, we propose three architectural improvements -- attention
  re-normalization, separable S^2 activation and separable layer normalization.
  Putting this all together, we propose EquiformerV2, which outperforms previous
  state-of-the-art methods on the large-scale OC20 dataset by up to 12% on
  forces, 4% on energies, offers better speed-accuracy trade-offs, and
  2times reduction in DFT calculations needed for computing adsorption
  energies.
paper_authors._id: 6493c9225b1cbc4829df2d9d
paper_authors.name: Yi-Lun Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-23T07:31:47.603Z
paper_authors._id: 64948a4a8d5ff0dd776655fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/65Z2SRYbzIXxraZE1e2Go.jpeg
paper_authors.fullname: Yi-Lun Liao
paper_authors.type: user
paper_authors.user: yilunliao
paper_authors._id: 6493c9225b1cbc4829df2d9e
paper_authors.name: Brandon Wood
paper_authors._id: 6493c9225b1cbc4829df2d9f
paper_authors.name: Abhishek Das
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-23T07:31:43.810Z
paper_authors._id: 6494e9e7b90702b268df53e3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6494e9e7b90702b268df53e3/hhAzJuQAJd1Gyv2XQFeks.jpeg
paper_authors.fullname: Abhishek Das
paper_authors.type: user
paper_authors.user: abhshkdz
paper_authors._id: 6493c9225b1cbc4829df2da0
paper_authors.name: Tess Smidt
paper_title: EquiformerV2: Improved Equivariant Transformer for Scaling to
    Higher-Degree Representations
paper_summary: Equivariant Transformers such as Equiformer have demonstrated the efficacy of
  applying Transformers to the domain of 3D atomistic systems. However, they are
  still limited to small degrees of equivariant representations due to their
  computational complexity. In this paper, we investigate whether these
  architectures can scale well to higher degrees. Starting from Equiformer, we
  first replace SO(3) convolutions with eSCN convolutions to efficiently
  incorporate higher-degree tensors. Then, to better leverage the power of higher
  degrees, we propose three architectural improvements -- attention
  re-normalization, separable S^2 activation and separable layer normalization.
  Putting this all together, we propose EquiformerV2, which outperforms previous
  state-of-the-art methods on the large-scale OC20 dataset by up to 12% on
  forces, 4% on energies, offers better speed-accuracy trade-offs, and
  2times reduction in DFT calculations needed for computing adsorption
  energies.
paper_ai_keywords: Equivariant Transformers Equiformer 3D atomistic systems $SO(3)$ convolutions eSCN convolutions higher-degree tensors attention re-normalization $S^2$ activation separable layer normalization EquiformerV2 OC20 dataset forces energies speed-accuracy trade-offs DFT calculations adsorption energies