Boosting X-formers with Structured Matrix for Long Sequence Time Series Forecasting
<|reference_start|>Boosting X-formers with Structured Matrix for Long Sequence Time Series Forecasting: Transformer-based models for long sequence time series forecasting (LSTF) problems have gained significant attention due to their exceptional forecasting precision. As the cornerstone of these models, the self-attention mechanism poses a challenge to efficient training and inference due to its quadratic time complexity. In this article, we propose a novel architectural design for Transformer-based models in LSTF, leveraging a substitution framework that incorporates Surrogate Attention Blocks and Surrogate FFN Blocks. The framework aims to boost any well-designed model's efficiency without sacrificing its accuracy. We further establish the equivalence of the Surrogate Attention Block to the self-attention mechanism in terms of both expressiveness and trainability. Through extensive experiments encompassing nine Transformer-based models across five time series tasks, we observe an average performance improvement of 9.45% while achieving a significant reduction in model size by 46%<|reference_end|>
@article{zhang2024boosting,
    title={Boosting X-formers with Structured Matrix for Long Sequence Time Series
  Forecasting},
    author={Zhicheng Zhang, Yong Wang, Shaoqi Tan, Bowei Xia, Yujie Luo},
    journal={arXiv preprint arXiv:2405.12462},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.12462},
    primaryClass={cs.LG cs.AI}
}