EquiformerV2: Improved Equivariant Transformer for Scaling to
  Higher-Degree Representations
Equivariant Transformers such as Equiformer have demonstrated the efficacy of
applying Transformers to the domain of 3D atomistic systems. However, they are
still limited to small degrees of equivariant representations due to their
computational complexity. In this paper, we investigate whether these
architectures can scale well to higher degrees. Starting from Equiformer, we
first replace SO(3) convolutions with eSCN convolutions to efficiently
incorporate higher-degree tensors. Then, to better leverage the power of higher
degrees, we propose three architectural improvements -- attention
re-normalization, separable S^2 activation and separable layer normalization.
Putting this all together, we propose EquiformerV2, which outperforms previous
state-of-the-art methods on the large-scale OC20 dataset by up to 12% on
forces, 4% on energies, offers better speed-accuracy trade-offs, and
2times reduction in DFT calculations needed for computing adsorption
energies.
6493c9225b1cbc4829df2d9d
Yi-Lun Liao
claimed_verified
2023-06-23T07:31:47.603Z
64948a4a8d5ff0dd776655fe
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/65Z2SRYbzIXxraZE1e2Go.jpeg
Yi-Lun Liao
user
yilunliao
6493c9225b1cbc4829df2d9e
Brandon Wood
6493c9225b1cbc4829df2d9f
Abhishek Das
claimed_verified
2023-06-23T07:31:43.810Z
6494e9e7b90702b268df53e3
https://cdn-avatars.huggingface.co/v1/production/uploads/6494e9e7b90702b268df53e3/hhAzJuQAJd1Gyv2XQFeks.jpeg
Abhishek Das
user
abhshkdz
6493c9225b1cbc4829df2da0
Tess Smidt
EquiformerV2: Improved Equivariant Transformer for Scaling to
  Higher-Degree Representations
Equivariant Transformers such as Equiformer have demonstrated the efficacy of
applying Transformers to the domain of 3D atomistic systems. However, they are
still limited to small degrees of equivariant representations due to their
computational complexity. In this paper, we investigate whether these
architectures can scale well to higher degrees. Starting from Equiformer, we
first replace SO(3) convolutions with eSCN convolutions to efficiently
incorporate higher-degree tensors. Then, to better leverage the power of higher
degrees, we propose three architectural improvements -- attention
re-normalization, separable S^2 activation and separable layer normalization.
Putting this all together, we propose EquiformerV2, which outperforms previous
state-of-the-art methods on the large-scale OC20 dataset by up to 12% on
forces, 4% on energies, offers better speed-accuracy trade-offs, and
2times reduction in DFT calculations needed for computing adsorption
energies.
Equivariant Transformers, specifically EquiformerV2, achieve superior performance in 3D atomistic systems by using eSCN convolutions and architectural improvements, demonstrating better scalability, speed, and accuracy.
Equivariant Transformers Equiformer $SO(3)$ convolutions eSCN convolutions higher-degree tensors attention re-normalization separable $S^2$ activation separable layer normalization EquiformerV2 OC20 dataset forces energies DFT calculations adsorption energies