summaries: This paper studies long-horizon time series forecasting where the historical input length is shorter than the forecast horizon, i.e, output length. This approach extends the transformer by replacing the self-attention unit by autocorrelation and decomposition units.  The autocorrelation unit chooses the top $k$ highly correlated time lags from $L$ time lags between the query and key embeddings of a time series of length $L$. The normalized correlation of lag j between the query and key embeddings multiplied by the j-item rotated value embedding.  The decomposition block decomposes a time series to seasonal and trend components. The trend component is defined by using an average pooling, which is an average value of the neighbouring values. And the seasonal is then defined as the difference between the (observed) time series and the trend component. The proposed approach autocorrelation unit is compared with standard attention, logsparse attention, LSH and probsparse attention.   This paper studies the problem of long-term forecasting for time series. There are two challenges: (1) it is difficult to directly study the entangled temporal dependencies for long time series; (2) the traditional self-attention mechanism in Transformer is computationally expensive. To address these two problems, the paper proposes a novel Autoformer with a novel auto-correlation mechanism and a series decomposition module.  This paper presents Autoformer to perform long-term time series forecasting. The key idea is to leverage an auto-correlation mechanism to discover the sub-series similarity based on the series periodicity and aggregate similar sub-series from underlying periods. The experiment results on several datasets showed the effectiveness of the proposed method. This paper focuses on the long-term forecasting problem of time series. This work introduces the traditional idea of decomposing time series into seasonality and trend-cycle components into deep Transformer architecture. This work also proposes an auto-correlation mechanism that replaces the original self-attention in Transformers.  Further, they improve the computation efficiency by computing power spectral density. Experiments on several datasets show that the proposed method achieves better performance compared to SOTA methods. This paper introduces a new approach for long-term time-series forecasting that is based on decomposition and auto-correlation. Essentially, the authors define a series decomoposition block which extracts the mean (trend-cyclical) and subtracts it from the series, obtaining the seasonal part. In addition, the authors design an auto-correlation component that identifies local structures across periods. The method is evaluated on six benchmark datasets and in comparison to several baseline approaches. The presented results show improvements over existing work.
abstractText: Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformerbased models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields stateof-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: https://github.com/thuml/Autoformer.
authors.name: Haixu Wu
authors.name: Jiehui Xu
authors.name: Jianmin Wang
authors.name: Mingsheng Long
references.authors: O. Anderson M. Kendall
references.title: Time-series. 2nd edn
references.venue: J. R. Stat. Soc. (Series D),
references.authors: Reza Asadi Amelia C Regan
references.title: A spatio-temporal decomposition based deep neural network for time series forecasting
references.venue: Appl. Soft Comput.,
references.authors: Shaojie Bai J Zico Kolter Vladlen Koltun
references.title: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling
references.venue: arXiv preprint arXiv:1803.01271,
references.authors: Anastasia Borovykh Sander Bohte Cornelis W Oosterlee
references.title: Conditional time series forecasting with convolutional neural networks
references.venue: arXiv preprint arXiv:1703.04691,
references.authors: G.E.P. Box Gwilym M. Jenkins
references.title: Time series analysis, forecasting and control
references.authors: George EP Box Gwilym M Jenkins
references.title: Some recent advances in forecasting and control
references.venue: J. R. Stat. Soc. (Series-C),
references.authors: Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel Ziegler Jeffrey Wu Clemens Winter Chris Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei
references.title: Language models are few-shot learners
references.venue: NeurIPS,
references.authors: Chris Chatfield
references.title: The analysis of time series: an introduction
references.authors: Renyi Chen Molei Tao
references.title: Data-driven prediction of general hamiltonian dynamics via learning exactlysymplectic maps
references.authors: Emmanuel de Bézenac Syama Sundar Rangapuram Konstantinos Benidis Michael Bohlke-Schneider Richard Kurle Lorenzo Stella Hilaf Hasson Patrick Gallinari Tim Januschowski
references.title: Normalizing kalman filters for multivariate time series analysis
references.venue: NeurIPS,
references.authors: J. Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.venue: In NAACL-HLT,
references.authors: Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly Jakob Uszkoreit Neil Houlsby
references.title: An image is worth 16x16 words: Transformers for image recognition at scale
references.venue: In ICLR,
references.authors: S. Hochreiter J. Schmidhuber
references.title: Long short-term memory
references.venue: Neural Comput.,
references.authors: Rob J Hyndman George Athanasopoulos
references.title: Forecasting: principles and practice
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In ICLR,
references.authors: Nikita Kitaev Lukasz Kaiser Anselm Levskaya
references.title: Reformer: The efficient transformer
references.venue: In ICLR,
references.authors: Richard Kurle Syama Sundar Rangapuram Emmanuel de Bézenac Stephan Günnemann Jan Gasthaus
references.title: Deep rao-blackwellised particle filters for time series forecasting
references.venue: NeurIPS,
references.authors: Guokun Lai Wei-Cheng Chang Yiming Yang Hanxiao Liu
references.title: Modeling long-and short-term temporal patterns with deep neural networks
references.venue: In SIGIR,
references.authors: Shiyang Li Xiaoyong Jin Yao Xuan Xiyou Zhou Wenhu Chen Yu-Xiang Wang Xifeng Yan
references.title: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting
references.venue: NeurIPS,
references.authors: Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin Baining Guo
references.title: Swin transformer: Hierarchical vision transformer using shifted windows
references.venue: In ICCV,
references.authors: Danielle C Maddix Yuyang Wang Alex Smola
references.title: Deep factors with gaussian processes for forecasting
references.venue: arXiv preprint arXiv:1812.00098,
references.authors: Boris N Oreshkin Dmitri Carpov Nicolas Chapados Yoshua Bengio
references.title: N-BEATS: Neural basis expansion analysis for interpretable time series forecasting
references.authors: Athanasios Papoulis H Saunders
references.title: Probability, random variables and stochastic processes
references.authors: Adam Paszke S. Gross Francisco Massa A. Lerer James Bradbury Gregory Chanan Trevor Killeen Z. Lin N. Gimelshein L. Antiga Alban Desmaison Andreas Köpf Edward Yang Zach DeVito Martin Raison Alykhan Tejani Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: NeurIPS,
references.authors: Syama Sundar Rangapuram Matthias W Seeger Jan Gasthaus Lorenzo Stella Yuyang Wang Tim Januschowski
references.title: Deep state space models for time series forecasting
references.venue: NeurIPS,
references.authors: Cleveland Robert C William Terpenning Irma
references.title: STL: A seasonal-trend decomposition procedure based on loess
references.venue: J. Off. Stat,
references.authors: David Salinas Valentin Flunkert Jan Gasthaus Tim Januschowski
references.title: DeepAR: Probabilistic forecasting with autoregressive recurrent networks
references.venue: Int. J. Forecast.,
references.authors: Rajat Sen Hsiang-Fu Yu Inderjit S. Dhillon
references.title: Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting
references.venue: NeurIPS,
references.authors: Shun-Yao Shih Fan-Keng Sun Hung-yi Lee
references.title: Temporal pattern attention for multivariate time series forecasting
references.authors: Huan Song Deepta Rajan Jayaraman Thiagarajan Andreas Spanias
references.title: Attend and diagnose: Clinical time series analysis using attention models
references.venue: In AAAI,
references.authors: Antti Sorjamaa Jin Hao Nima Reyhani Yongnan Ji Amaury Lendasse
references.title: Methodology for long-term prediction of time series
references.authors: Sean J Taylor Benjamin Letham
references.title: Forecasting at scale
references.venue: Am. Stat.,
references.authors: Aäron van den Oord S. Dieleman H. Zen K. Simonyan Oriol Vinyals A. Graves Nal Kalchbrenner A. Senior K. Kavukcuoglu
references.title: Wavenet: A generative model for raw audio
references.venue: SSW,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Ł ukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: NeurIPS,
references.authors: Ruofeng Wen Kari Torkkola Balakrishnan Narayanaswamy Dhruv Madeka
references.title: A multi-horizon quantile recurrent forecaster
references.venue: NeurIPS,
references.authors: Norbert Wiener
references.title: Generalized harmonic analysis
references.venue: Acta Math,
references.authors: Sifan Wu Xi Xiao Qianggang Ding Peilin Zhao Ying Wei Junzhou Huang
references.title: Adversarial sparse transformer for time series forecasting
references.venue: NeurIPS,
references.authors: Q. Yao D. Song H. Chen C. Wei G.W. Cottrell
references.title: A dual-stage attention-based recurrent neural network for time series prediction
references.authors: Rose Yu Stephan Zheng Anima Anandkumar Yisong Yue
references.title: Long-term forecasting using tensor-train rnns
references.venue: arXiv preprint arXiv:1711.00073,
references.authors: Haoyi Zhou Shanghang Zhang Jieqi Peng Shuai Zhang Jianxin Li Hui Xiong Wancai Zhang
references.title: Informer: Beyond efficient transformer for long sequence time-series forecasting
references.venue: In AAAI,
sections.heading: 1 Introduction
sections.text: Time series forecasting has been widely used in energy consumption, traffic and economics planning, weather and disease propagation forecasting. In these real-world applications, one pressing demand is to extend the forecast time into the far future, which is quite meaningful for the long-term planning and early warning. Thus, in this paper, we study the long-term forecasting problem of time series, characterizing itself by the large length of predicted time series. Recent deep forecasting models [41, 17, 20, 28, 23, 29, 19, 35] have achieved great progress, especially the Transformer-based models. Benefiting from the self-attention mechanism, Transformers obtain great advantage in modeling long-term dependencies for sequential data, which enables more powerful big models [7, 11].
  However, the forecasting task is extremely challenging under the long-term setting. First, it is unreliable to discover the temporal dependencies directly from the long-term time series because the dependencies can be obscured by entangled temporal patterns. Second, canonical Transformers with self-attention mechanisms are computationally prohibitive for long-term forecasting because of the quadratic complexity of sequence length. Previous Transformer-based forecasting models [41, 17, 20] mainly focus on improving self-attention to a sparse version. While performance is significantly improved, these models still utilize the point-wise representation aggregation. Thus, in the process of efficiency improvement, they will sacrifice the information utilization because of the sparse point-wise connections, resulting in a bottleneck for long-term forecasting of time series.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  To reason about the intricate temporal patterns, we try to take the idea of decomposition, which is a standard method in time series analysis [1, 27]. It can be used to process the complex time series and extract more predictable components. However, under the forecasting context, it can only be used as the pre-processing of past series because the future is unknown [15]. This common usage limits the capabilities of decomposition and overlooks the potential future interactions among decomposed components. Thus, we attempt to go beyond pre-processing usage of decomposition and propose a generic architecture to empower the deep forecasting models with immanent capacity of progressive decomposition. Further, decomposition can ravel out the entangled temporal patterns and highlight the inherent properties of time series [15]. Benefiting from this, we try to take advantage of the series periodicity to renovate the point-wise connection in self-attention. We observe that the sub-series at the same phase position among periods often present similar temporal processes. Thus, we try to construct a series-level connection based on the process similarity derived by series periodicity.
  Based on the above motivations, we propose an original Autoformer in place of the Transformers for long-term time series forecasting. Autoformer still follows residual and encoder-decoder structure but renovates Transformer into a decomposition forecasting architecture. By embedding our proposed decomposition blocks as the inner operators, Autoformer can progressively separate the long-term trend information from predicted hidden variables. This design allows our model to alternately decompose and refine the intermediate results during the forecasting procedure. Inspired by the stochastic process theory [8, 24], Autoformer introduces an Auto-Correlation mechanism in place of self-attention, which discovers the sub-series similarity based on the series periodicity and aggregates similar sub-series from underlying periods. This series-wise mechanism achieves O(L logL) complexity for length-L series and breaks the information utilization bottleneck by expanding the point-wise representation aggregation to sub-series level. Autoformer achieves the state-of-the-art accuracy on six benchmarks. The contributions are summarized as follows:
  • To tackle the intricate temporal patterns of the long-term future, we present Autoformer as a decomposition architecture and design the inner decomposition block to empower the deep forecasting model with immanent progressive decomposition capacity.
  • We propose an Auto-Correlation mechanism with dependencies discovery and information aggregation at the series level. Our mechanism is beyond previous self-attention family and can simultaneously benefit the computation efficiency and information utilization.
  • Autoformer achieves a 38% relative improvement under the long-term setting on six benchmarks, covering five real-world applications: energy, traffic, economics, weather and disease.
sections.heading: 2 Related Work
sections.heading: 2.1 Models for Time Series Forecasting
sections.text: Due to the immense importance of time series forecasting, various models have been well developed. Many time series forecasting methods start from the classic tools [32, 9]. ARIMA [6, 5] tackles the forecasting problem by transforming the non-stationary process to stationary through differencing. The filtering method is also introduced for series forecasting [18, 10]. Besides, recurrent neural networks (RNNs) models are used to model the temporal dependencies for time series [36, 26, 40, 22]. DeepAR [28] combines autoregressive methods and RNNs to model the probabilistic distribution of future series. LSTNet [19] introduces convolutional neural networks (CNNs) with recurrent-skip connections to capture the short-term and long-term temporal patterns. Attention-based RNNs [39, 30, 31] introduce the temporal attention to explore the long-range dependencies for prediction. Also, many works based on temporal convolution networks (TCN) [34, 4, 3, 29] attempt to model the temporal causality with the causal convolution. These deep forecasting models mainly focus on the temporal relation modeling by recurrent connections, temporal attention or causal convolution.
  Recently, Transformers [35, 38] based on the self-attention mechanism shows great power in sequential data, such as natural language processing [11, 7], audio processing [14] and even computer vision [12, 21]. However, applying self-attention to long-term time series forecasting is computationally prohibitive because of the quadratic complexity of sequence length L in both memory and time. LogTrans [20] introduces the local convolution to Transformer and proposes the LogSparse attention to select time steps following the exponentially increasing intervals, which reduces the complexity to O(L(logL)2). Reformer [17] presents the local-sensitive hashing (LSH) attention and reduces the complexity to O(L logL). Informer [41] extends Transformer with KL-divergence based ProbSparse
  attention and also achieves O(L logL) complexity. Note that these methods are based on the vanilla Transformer and try to improve the self-attention mechanism to a sparse version, which still follows the point-wise dependency and aggregation. In this paper, our proposed Auto-Correlation mechanism is based on the inherent periodicity of time series and can provide series-wise connections.
sections.heading: 2.2 Decomposition of Time Series
sections.text: As a standard method in time series analysis, time series decomposition [1, 27] deconstructs a time series into several components, each representing one of the underlying categories of patterns that are more predictable. It is primarily useful for exploring historical changes over time. For the forecasting tasks, decomposition is always used as the pre-processing of historical series before predicting future series [15, 2], such as Prophet [33] with trend-seasonality decomposition and N-BEATS [23] with basis expansion and DeepGLO [29] with matrix decomposition. However, such pre-processing is limited by the plain decomposition effect of historical series and overlooks the hierarchical interaction between the underlying patterns of series in the long-term future. This paper takes the decomposition idea from a new progressive dimension. Our Autoformer harnesses the decomposition as an inner block of deep models, which can progressively decompose the hidden series throughout the whole forecasting process, including both the past series and the predicted intermediate results.
sections.heading: 3 Autoformer
sections.text: The time series forecasting problem is to predict the most probable length-O series in the future given the past length-I series, denoting as input-I-predict-O. The long-term forecasting setting is to predict the long-term future, i.e. larger O. As aforementioned, we have highlighted the difficulties of long-term series forecasting: handling intricate temporal patterns and breaking the bottleneck of computation efficiency and information utilization. To tackle these two challenges, we introduce the decomposition as a builtin block to the deep forecasting model and propose Autoformer as a decomposition architecture. Besides, we design the Auto-Correlation mechanism to discover the period-based dependencies and aggregate similar sub-series from underlying periods.
sections.heading: 3.1 Decomposition Architecture
sections.text: We renovate Transformer [35] to a deep decomposition architecture (Figure 1), including the inner series decomposition block, Auto-Correlation mechanism, and corresponding Encoder and Decoder.
  Series decomposition block To learn with the complex temporal patterns in long-term forecasting context, we take the idea of decomposition [1, 27], which can separate the series into trend-cyclical and seasonal parts. These two parts reflect the long-term progression and the seasonality of the series respectively. However, directly decomposing is unrealizable for future series because the future is just unknown. To tackle this dilemma, we present a series decomposition block as an inner operation of Autoformer (Figure 1), which can extract the long-term stationary trend from predicted intermediate hidden variables progressively. Concretely, we adapt the moving average to smooth out periodic fluctuations and highlight the long-term trends. For length-L input series X ∈ RL×d, the process is:
  Xt = AvgPool(Padding(X )) Xs = X − Xt,
  (1)
  where Xs,Xt ∈ RL×d denote the seasonal and the extracted trend-cyclical part respectively. We adopt the AvgPool(·) for moving average with the padding operation to keep the series length unchanged. We use Xs,Xt = SeriesDecomp(X ) to summarize above equations, which is a model inner block. Model inputs The inputs of encoder part are the past I time stepsXen ∈ RI×d. As a decomposition architecture (Figure 1), the input of Autoformer decoder contains both the seasonal part Xdes ∈ R( I2+O)×d and trend-cyclical part Xdet ∈ R( I 2+O)×d to be refined. Each initialization consists of two parts: the component decomposed from the latter half of encoder’s input Xen with length I2 to provide recent information, placeholders with length O filled by scalars. It’s formulized as follows:
  Xens,Xent = SeriesDecomp(Xen I 2 :I )
  Xdes = Concat(Xens,X0) Xdet = Concat(Xent,XMean),
  (2)
  where Xens,Xent ∈ R I 2×d denote the seasonal and trend-cyclical parts of Xen respectively, and X0,XMean ∈ RO×d denote the placeholders filled with zero and the mean of Xen respectively.
  Encoder As shown in Figure 1, the encoder focuses on the seasonal part modeling. The output of the encoder contains the past seasonal information and will be used as the cross information to help the decoder refine prediction results. Suppose we have N encoder layers. The overall equations for l-th encoder layer are summarized as X len = Encoder(X l−1en ). Details are shown as follows:
  Sl,1en , = SeriesDecomp ( Auto-Correlation(X l−1en ) + X l−1en ) Sl,2en , = SeriesDecomp ( FeedForward(Sl,1en ) + Sl,1en ) ,
  (3)
  where “ ” is the eliminated trend part. X len = Sl,2en , l ∈ {1, · · · , N} denotes the output of l-th encoder layer and X 0en is the embedded Xen. Sl,ien , i ∈ {1, 2} represents the seasonal component after the i-th series decomposition block in the l-th layer respectively. We will give detailed description of Auto-Correlation(·) in the next section, which can seamlessly replace the self-attention.
  Decoder The decoder contains two parts: the accumulation structure for trend-cyclical components and the stacked Auto-Correlation mechanism for seasonal components (Figure 1). Each decoder layer contains the inner Auto-Correlation and encoder-decoder Auto-Correlation, which can refine the prediction and utilize the past seasonal information respectively. Note that the model extracts the potential trend from the intermediate hidden variables during the decoder, allowing Autoformer to progressively refine the trend prediction and eliminate interference information for period-based dependencies discovery in Auto-Correlation. Suppose there are M decoder layers. With the latent variable XNen from the encoder, the equations of l-th decoder layer can be summarized as X lde = Decoder(X l−1de ,XNen). The decoder can be formalized as follows:
  Sl,1de , T l,1 de = SeriesDecomp ( Auto-Correlation(X l−1de ) + X l−1de ) Sl,2de , T l,2 de = SeriesDecomp ( Auto-Correlation(Sl,1de ,XNen) + S l,1 de
  ) Sl,3de , T l,3 de = SeriesDecomp ( FeedForward(Sl,2de ) + S l,2 de
  ) T lde = T l−1de +Wl,1 ∗ T l,1 de +Wl,2 ∗ T l,2 de +Wl,3 ∗ T l,3 de ,
  (4)
  where X lde = Sl,3de , l ∈ {1, · · · ,M} denotes the output of l-th decoder layer. X 0de is embedded from Xdes for deep transform and T 0de = Xdet is for accumulation. Sl,ide , T l,i de , i ∈ {1, 2, 3} represent the seasonal component and trend-cyclical component after the i-th series decomposition block in the l-th layer respectively. Wl,i, i ∈ {1, 2, 3} represents the projector for the i-th extracted trend T l,ide .
  … L
  Q K V
  FFT
  x
  FFT
  Inverse FFT
  Time Delay Agg
  Linear Linear Linear
  Linear
  Concat
  LxC SxC SxC
  LxCx2
  LxC
  LxC
  LxCx2 ConjugateLxCx2
  LxC
  Resize Resize
  LxC
  Roll( )
  Roll( )
  Roll( )
  x
  x
  x
  …
  Fu sio n Time Delay Top k kxC SoftMax
  test
  haixu wu
  April 2021
  1 Introduction
  ⌧1 ⌧2 ⌧k R(⌧1) R(⌧2) R(⌧k)
  1
  test
  haixu wu
  April 2021
  1 Introduction
  ⌧1 ⌧2 ⌧k R(⌧1) R(⌧2) R(⌧k)
  1
  test
  haixu wu
  April 2021
  1 Introduction
  ⌧1 2
  ⌧k ⌧1) ⌧2) R(⌧k)
  1
  test
  haixu wu
  April 2021
  1 Introduction
  ⌧1 ⌧2 ⌧k R(⌧1) R(⌧2) R(⌧k)
  1
  test
  haixu wu
  April 2021
  1 Introduction
  ⌧1 ⌧2 ⌧k
  (⌧1) R(⌧2) R(⌧k)
  1
  test
  haixu wu
  April 2021
  1 Introduction
  ⌧1 ⌧2 ⌧k R(⌧1) R(⌧2) R(⌧k)
  1
  Figure 2: Auto-Correlation (left) and Time Delay Aggregation (right). We utilize the Fast Fourier Transform to calculate the autocorrelationR(τ), which reflects the time-delay similarities. Then the similar sub-processes are rolled to the same index based on selected delay τ and aggregated byR(τ).
  The final prediction is the sum of the two refined decomposed components, asWS ∗ XMde + T Mde , whereWS is to project the deep transformed seasonal component XMde to the target dimension.
sections.heading: 3.2 Auto-Correlation Mechanism
sections.text: As shown in Figure 2, we propose the Auto-Correlation mechanism with series-wise connections to expand the information utilization. Auto-Correlation discovers the period-based dependencies by calculating the series autocorrelation and aggregates similar sub-series by time delay aggregation.
  Period-based dependencies It is observed that the same phase position among periods naturally provides similar sub-processes. Inspired by the stochastic process theory [8, 24], for a real discretetime process {Xt}, we can obtain the autocorrelationRXX (τ) by the following equations:
  RXX (τ) = lim L→∞
  1
  L L∑ t=1 XtXt−τ . (5)
  RXX (τ) reflects the time-delay similarity between {Xt} and its τ lag series {Xt−τ}. As shown in Figure 2, we use the autocorrelationR(τ) as the unnormalized confidence of estimated period length τ . Then, we choose the most possible k period lengths τ1, · · · , τk. The period-based dependencies are derived by the above estimated periods and can be weighted by the corresponding autocorrelation.
  Time delay aggregation The period-based dependencies connect the sub-series among estimated periods. Thus, we present the time delay aggregation block (Figure 2), which can roll the series based on selected time delay τ1, · · · , τk. This operation can align similar sub-series that are at the same phase position of estimated periods, which is different from the point-wise dot-product aggregation in self-attention family. Finally, we aggregate the sub-series by softmax normalized confidences.
  For the single head situation and time series X with length-L, after the projector, we get queryQ, key K and value V . Thus, it can replace self-attention seamlessly. The Auto-Correlation mechanism is:
  τ1, · · · , τk = argTopk τ∈{1,··· ,L} (RQ,K(τ))
  R̂Q,K(τ1), · · · , R̂Q,K(τk) = SoftMax (RQ,K(τ1), · · · ,RQ,K(τk))
  Auto-Correlation(Q,K,V) = k∑ i=1 Roll(V, τi)R̂Q,K(τi),
  (6)
  where arg Topk(·) is to get the arguments of the Topk autocorrelations and let k = bc× logLc, c is a hyper-parameter. RQ,K is autocorrelation between series Q and K. Roll(X , τ) represents the operation to X with time delay τ , during which elements that are shifted beyond the first position are re-introduced at the last position. For the encoder-decoder Auto-Correlation (Figure 1), K,V are from the encoder XNen and will be resized to length-O, Q is from the previous block of the decoder.
  For the multi-head version used in Autoformer, with hidden variables of dmodel channels, h heads, the query, key and value for i-th head are Qi,Ki,Vi ∈ RL× dmodel h , i ∈ {1, · · · , h}. The process is:
  MultiHead(Q,K,V) =Woutput ∗ Concat(head1, · · · ,headh) where headi = Auto-Correlation(Qi,Ki,Vi).
  (7)
  Efficient computation For period-based dependencies, these dependencies point to sub-processes at the same phase position of underlying periods and are inherently sparse. Here, we select the most possible delays to avoid picking the opposite phases. Because we aggregate O(logL) series whose length is L, the complexity of Equations 6 and 7 is O(L logL). For the autocorrelation computation (Equation 5), given time series {Xt},RXX (τ) can be calculated by Fast Fourier Transforms (FFT) based on the Wiener–Khinchin theorem [37]:
  SXX (f) = F (Xt)F∗ (Xt) = ∫ ∞ −∞ Xte−i2πtfdt ∫ ∞ −∞ Xte−i2πtfdt
  RXX (τ) = F−1 (SXX (f)) = ∫ ∞ −∞ SXX (f)ei2πfτdf,
  (8)
  where τ ∈ {1, · · · , L}, F denotes the FFT and F−1 is its inverse. ∗ denotes the conjugate operation and SXX (f) is in the frequency domain. Note that the series autocorrelation of all lags in {1, · · · , L} can be calculated at once by FFT. Thus, Auto-Correlation achieves the O(L logL) complexity.
  Auto-Correlation vs. self-attention family Different from the point-wise self-attention family, Auto-Correlation presents the series-wise connections (Figure 3). Concretely, for the temporal dependencies, we find the dependencies among sub-series based on the periodicity. In contrast, the self-attention family only calculates the relation between scattered points. Though some selfattentions [20, 41] consider the local information, they only utilize this to help point-wise dependencies discovery. For the information aggregation, we adopt the time delay block to aggregate the similar sub-series from underlying periods. In contrast, self-attentions aggregate the selected points by dot-product. Benefiting from the inherent sparsity and sub-series-level representation aggregation, Auto-Correlation can simultaneously benefit the computation efficiency and information utilization.
sections.heading: 4 Experiments
sections.text: We extensively evaluate the proposed Autoformer on six real-world benchmarks, covering five mainstream time series forecasting applications: energy, traffic, economics, weather and disease.
  Datasets Here is a description of the six experiment datasets: (1) ETT [41] dataset contains the data collected from electricity transformers, including load and oil temperature that are recorded every
  15 minutes between July 2016 and July 2018. (2) Electricity1 dataset contains the hourly electricity consumption of 321 customers from 2012 to 2014. (3) Exchange [19] records the daily exchange rates of eight different countries ranging from 1990 to 2016. (4) Traffic2 is a collection of hourly data from California Department of Transportation, which describes the road occupancy rates measured by different sensors on San Francisco Bay area freeways. (5) Weather3 is recorded every 10 minutes for 2020 whole year, which contains 21 meteorological indicators, such as air temperature, humidity, etc. (6) ILI4 includes the weekly recorded influenza-like illness (ILI) patients data from Centers for Disease Control and Prevention of the United States between 2002 and 2021, which describes the ratio of patients seen with ILI and the total number of the patients. We follow standard protocol and split all datasets into training, validation and test set in chronological order by the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets.
  Implementation details Our method is trained with L2 loss, using the ADAM [16] optimizer with an initial learning rate of 10−4. Batch size is set to 32. The training process is early stopped within 10 epochs. All experiments are repeated three times, implemented in PyTorch [25] and conducted on a single NVIDIA TITAN RTX 24GB GPUs. The hyper-parameter c of Auto-Correlation is in the range of 1 to 3 to trade off performance and efficiency. See supplementary materials for standard deviations and sensitivity analysis. Autoformer contains 2 encoder layers and 1 decoder layer.
  Baselines We include 10 baseline methods. For the multivariate setting, we select three latest stateof-the-art transformer-based models: Informer [41], Reformer [17], LogTrans [20], two RNN-based models: LSTNet [19], LSTM [13] and CNN-based TCN [3] as baselines. For the univariate setting, we include more competitive baselines: N-BEATS[23], DeepAR [28], Prophet [33] and ARMIA [1].
  1https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014 2http://pems.dot.ca.gov 3https://www.bgc-jena.mpg.de/wetter/ 4https://gis.cdc.gov/grasp/fluview/fluportaldashboard.html
sections.heading: 4.1 Main Results
sections.text: To compare performances under different future horizons, we fix the input length and evaluate models with a wide range of prediction lengths: 96, 192, 336, 720. This setting precisely meets the definition of long-term forecasting. Here are results on both the multivariate and univariate settings.
  Multivariate results As for the multivariate setting, Autoformer achieves the consistent state-ofthe-art performance in all benchmarks and all prediction length settings (Table 1). Especially, under the input-96-predict-336 setting, compared to previous state-of-the-art results, Autoformer gives 74% (1.334→0.339) MSE reduction in ETT, 18% (0.280→0.231) in Electricity, 61% (1.357→0.509) in Exchange, 15% (0.733→0.622) in Traffic and 21% (0.455→0.359) in Weather. For the input36-predict-60 setting of ILI, Autoformer makes 43% (4.882→2.770) MSE reduction. Overall, Autoformer yields a 38% averaged MSE reduction among above settings. Note that Autoformer still provides remarkable improvements in the Exchange dataset that is without obvious periodicity. See supplementary materials for detailed showcases. Besides, we can also find that the performance of Autoformer changes quite steadily as the prediction length O increases. It means that Autoformer retains better long-term robustness, which is meaningful for real-world practical applications, such as weather early warning and long-term energy consumption planning.
  Univariate results We list the univariate results of two typical datasets in Table 2. Under the comparison with extensive baselines, our Autoformer still achieves state-of-the-art performance for the long-term forecasting tasks. In particular, for the input-96-predict-336 setting, our model achieves 14% (0.180→0.145) MSE reduction on the ETT dataset with obvious periodicity. For the Exchange dataset without obvious periodicity, Autoformer surpasses other baselines by 17% (0.611→0.508) and shows greater long-term forecasting capacity. Also, we find that ARIMA [1] performs best in the input-96-predict-96 setting of the Exchange dataset but fails in the long-term setting. This situation of ARIMA can be benefited from its inherent capacity for non-stationary economic data but is limited by the intricate temporal patterns of real-world series.
sections.heading: 4.2 Ablation studies
sections.text: Decomposition architecture With our proposed progressive decomposition architecture, other models can gain consistent promotion, especially as the prediction length O increases (Table 3). This
  verifies that our method can generalize to other models and release the capacity of other dependencies learning mechanisms, alleviate the distraction caused by intricate patterns. Besides, our architecture outperforms the pre-processing, although the latter employs a bigger model and more parameters. Especially, pre-decomposing may even bring negative effect because it neglects the interaction of components during long-term future, such as Transformer [35] predict-720, Informer [41] predict-336.
  Auto-Correlation vs. self-attention family As shown in Table 4, our proposed Auto-Correlation achieves the best performance under various input-I-predict-O settings, which verifies the effectiveness of series-wise connections comparing to point-wise self-attentions (Figure 3). Furthermore, we can also observe that Auto-Correlation is memory efficiency from the last column of Table 4, which can be used in long sequence forecasting, such as input-336-predict-1440.
sections.heading: 4.3 Model Analysis
sections.text: Time series decomposition As shown in Figure 4, without our series decomposition block, the forecasting model cannot capture the increasing trend and peaks of the seasonal part. By adding the series decomposition blocks, Autoformer can aggregate and refine the trend-cyclical part from series progressively. This design also facilitates the learning of the seasonal part, especially the peaks and troughs. This verifies the necessity of our proposed progressive decomposition architecture.
  Dependencies learning The marked time delay sizes in Figure 5(a) indicate the most likely periods. Our learned periodicity can guide the model to aggregate the sub-series from the same or neighbor phase of periods by Roll(X , τi), i ∈ {1, · · · , 6}. For the last time step (declining stage), AutoCorrelation fully utilizes all similar sub-series without omissions or errors compared to self-attentions. This verifies that Autoformer can discover the relevant information more sufficiently and precisely.
  Complex seasonality modeling As shown in Figure 6, the lags that Autoformer learns from deep representations can indicate the real seasonality of raw series. For example, the learned lags of the
  daily recorded Exchange dataset present the monthly, quarterly and yearly periods (Figure 6 (b)). For the hourly recorded Traffic dataset (Figure 6 (c)), the learned lags show the intervals as 24-hours and 168-hours, which match the daily and weekly periods of real-world scenarios. These results show that Autoformer can capture the complex seasonalities of real-world series from deep representations and further provide a human-interpretable prediction.
  Efficiency analysis We compare the running memory and time among Auto-Correlation-based and self-attention-based models (Figure 7) during the training phase. The proposed Autoformer shows O(L logL) complexity in both memory and time and achieves better long-term sequences efficiency.
sections.heading: 5 Conclusions
sections.text: This paper studies the long-term forecasting problem of time series, which is a pressing demand for real-world applications. However, the intricate temporal patterns prevent the model from learning reliable dependencies. We propose the Autoformer as a decomposition architecture by embedding the series decomposition block as an inner operator, which can progressively aggregate the longterm trend part from intermediate prediction. Besides, we design an efficient Auto-Correlation mechanism to conduct dependencies discovery and information aggregation at the series level, which contrasts clearly from the previous self-attention family. Autoformer can naturally achieveO(L logL) complexity and yield consistent state-of-the-art performance in extensive real-world datasets.
  Acknowledgments and Disclosure of Funding
  This work was supported by the National Natural Science Foundation of China under Grants 62022050 and 62021002, Beijing Nova Program under Grant Z201100006820041, China’s Ministry of Industry and Information Technology, the MOE Innovation Plan and the BNRist Innovation Fund.
title: Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting