SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity
<|reference_start|>SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity: Learning representations on large graphs is a long-standing challenge due to the inter-dependence nature. Transformers recently have shown promising performance on small graphs thanks to its global attention for capturing all-pair interactions beyond observed structures. Existing approaches tend to inherit the spirit of Transformers in language and vision tasks, and embrace complicated architectures by stacking deep attention-based propagation layers. In this paper, we attempt to evaluate the necessity of adopting multi-layer attentions in Transformers on graphs, which considerably restricts the efficiency. Specifically, we analyze a generic hybrid propagation layer, comprised of all-pair attention and graph-based propagation, and show that multi-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs, particularly through simplifying model architectures without sacrificing expressiveness. As exemplified by this work, we propose a Simplified Single-layer Graph Transformers (SGFormer), whose main component is a single-layer global attention that scales linearly w.r.t. graph sizes and requires none of any approximation for accommodating all-pair interactions. Empirically, SGFormer successfully scales to the web-scale graph ogbn-papers100M, yielding orders-of-magnitude inference acceleration over peer Transformers on medium-sized graphs, and demonstrates competitiveness with limited labeled data.<|reference_end|>
@article{wu2024sgformer:,
    title={SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear
  Complexity},
    author={Qitian Wu, Kai Yang, Hengrui Zhang, David Wipf, Junchi Yan},
    journal={arXiv preprint arXiv:2409.09007},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.09007},
    primaryClass={cs.LG cs.AI}
}