summaries: For multivariate time series forecasting, this paper proposes to use a tensor network to model the variable space and improve the quality of the variable space by designing the series-variable encoder. Under the variable space, this paper also proposes an N-order residual connection approach and the skip-connection layer for processing the long-term data. The proposed model MVSRTN achieves good results on some datasets. However, this paper has not well explored MVSRTN, and the results are not very competitive. In addition, there are many typos in the paper. The paper proposes a deep learning architecture for time series forecasting called MVSRTN. The model is composed of 3 blocks with skip-connections in-between them: a 1) "Series-Variable Encoder", 2) an "N-Order Residual Tensor Network", as the authors call it, and 3) the output layer. Out of these, 1) is essentially a 1D CNN combined with (causal and non-causal) self-attention, and 2) seems to be a tensor network built from taking the tensor product of the sequence entries, and then contracting that by a TT-rank-constrained weight tensor. The computation of the tensor network output is formulated as a recursion across time-steps, and motivated by residual networks, an identity mapping with respect to the hidden state at the previous time-stamp is added to the recursive formulation. Then, by the analogy with higher-order solvers for ODEs, a "higher-order residual connection" is introduced. The model is applied to four time series forecasting datasets, where some improvements are achieved, and an ablation study is included to show how the various model blocks contribute to the performance. This paper proposed the MVSRTN architecture for multivariate time series modeling. The MVSRTN consists of an encoder to extract latent variables and residual tensor network (TN) blocks to capture the interactions in the latent space.  The main contribution of the model is the TN block part. In particular, the authors used tensor-products to fully represent the latent variable space (Eq 2). Then, they proposed an N-order residual TN block to alleviate potential gradient problems of high-order TNs in long-term time series. They conducted experiments on four multivariate time series data for prediction. Moreover, an ablation study shows the effectiveness of the proposed residual TN blocks.  Combining ResNet and TN is an interesting direction. However, I think this paper did not present this problem sufficiently and some notations are confusing.  The paper considers the problem of forecasting of multivariate time series. The authors propose an architecture for such forecasting that incorporates several layers, including a residual tensor network layer. The idea is to tensorize (via outer products/Kronecker products) the features passed on from the encoder layer and then handle these very large tensorized features with a tensor network. The point of the tensorization is to better incorporate the effect of combinations of variables from different time steps. Experiments are done on four benchmark datasets.
abstractText: Multivariate time series involve a series of valuable applications in the real world, and the basic premise of which is that multiple variables are interdependent. However, the relationship between variables in the latent space is dynamic and complex, and as the time window increases, the size of the space also increases exponentially. For fully exploiting the dependencies in the variable space, we propose Modeling Variable Space with Residual Tensor Networks (MVSRTN) for multivariate time series. In this framework, we derive the mathematical representation of the variable space, and then use a tensor network based on the idea of low-rank approximation to model the variable space. The tensor components are shared to ensure the translation invariance of the network. In order to improve the ability to model long-term sequences, we propose an N-order residual connection approach and couple it to the space-approximated tensor network. Moreover, the seriesvariable encoder is designed to improve the quality of the variable space, and we use the skip-connection layer to achieve the dissemination of information such as scale. Experimental results verify the effectiveness of our proposed method on four multivariate time series forecasting benchmark datasets.
references.authors: George EP Box Gwilym M Jenkins Gregory C Reinsel Greta M Ljung
references.title: Time series analysis: forecasting and control
references.authors: Jiezhu Cheng Kaizhu Huang Zibin Zheng
references.title: Towards better forecasting by fusing near and distant future visions
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Thomas Demeester
references.title: System identification with time-aware neural sequence models
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Roger Frigola
references.title: Bayesian time series learning with Gaussian processes
references.venue: PhD thesis, University of Cambridge,
references.authors: Xavier Glorot Yoshua Bengio
references.title: Understanding the difficulty of training deep feedforward neural networks. volume
references.venue: Proceedings of Machine Learning Research,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: Dan Hendrycks Kevin Gimpel
references.title: Gaussian error linear units (gelus)
references.venue: arXiv preprint arXiv:1606.08415,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Guokun Lai Wei-Cheng Chang Yiming Yang Hanxiao Liu
references.title: Modeling long-and short-term temporal patterns with deep neural networks
references.venue: In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval,
references.authors: Moshe Leshno Vladimir Ya Lin Allan Pinkus Shimon Schocken
references.title: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function
references.venue: Neural networks,
references.authors: Bei Li Quan Du Tao Zhou Shuhan Zhou Xin Zeng Tong Xiao Jingbo Zhu
references.title: Ode transformer: An ordinary differential equation-inspired model for neural machine translation
references.venue: arXiv preprint arXiv:2104.02308,
references.authors: Xindian Ma Peng Zhang Shuai Zhang Nan Duan Yuexian Hou Ming Zhou Dawei Song
references.title: A tensorized transformer for language modeling
references.venue: Advances in Neural Information Processing Systems,
references.authors: Jacob Miller Guillaume Rabusseau John Terilla
references.title: Tensor networks for probabilistic sequence modeling
references.venue: In International Conference on Artificial Intelligence and Statistics,
references.authors: Vinod Nair Geoffrey E Hinton
references.title: Rectified linear units improve restricted boltzmann machines
references.venue: In Icml,
references.authors: I.V. Oseledets
references.title: Tensor-train decomposition
references.venue: SIAM Journal on Scientific Computing,
references.authors: Syama Sundar Rangapuram Matthias W Seeger Jan Gasthaus Lorenzo Stella Yuyang Wang Tim Januschowski
references.title: Deep state space models for time series forecasting
references.venue: Advances in neural information processing systems,
references.authors: S. Roberts M. Osborne M. Ebden S. Reece N. Gibson S. Aigrain
references.title: Gaussian processes for time-series modelling
references.venue: Philosophical Transactions,
references.authors: Tim Salimans Durk P Kingma
references.title: Weight normalization: A simple reparameterization to accelerate training of deep neural networks
references.venue: In Advances in neural information processing systems,
references.authors: Shun-Yao Shih Fan-Keng Sun Hung-yi Lee
references.title: Temporal pattern attention for multivariate time series forecasting
references.venue: Machine Learning,
references.authors: Edwin Stoudenmire David J Schwab
references.title: Supervised learning with tensor networks
references.venue: Advances in Neural Information Processing Systems,
references.authors: Zonghan Wu Shirui Pan Guodong Long Jing Jiang Xiaojun Chang Chengqi Zhang
references.title: Connecting the dots: Multivariate time series forecasting with graph neural networks
references.venue: In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,
references.authors: Dongkuan Xu Wei Cheng Bo Zong Dongjin Song Jingchao Ni Wenchao Yu Yanchi Liu Haifeng Chen Xiang Zhang
references.title: Tensorized lstm with adaptive shared memory for learning trends in multivariate time series
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Valentin Zauner-Stauber Laurens Vanderstraeten Matthew T Fishman Frank Verstraete Jutho Haegeman
references.title: Variational optimization algorithms for uniform matrix product states
references.venue: Physical Review B,
references.authors: Lipeng Zhang Peng Zhang Xindian Ma Shuqin Gu Zhan Su Dawei Song
references.title: A generalized language model in tensor space
references.venue: In Proceedings of the AAAI Conference on Artificial Intelligence,
references.authors: Hang Zhao Yujing Wang Juanyong Duan Congrui Huang Defu Cao Yunhai Tong Bixiong Xu Jing Bai Jie Tong Qi Zhang
references.title: Multivariate time-series anomaly detection via graph attention network
references.venue: IEEE International Conference on Data Mining (ICDM),
references.authors: Mai Zhu Bo Chang Chong Fu
references.title: Convolutional neural networks combined with runge-kutta methods, 2019
references.venue: URL https://openreview.net/forum?id=HJNJws0cF7
sections.heading: 1 INTRODUCTION
sections.text: Multivariate time series are ubiquitous in the real world, such as energy collection and consumption (Rangapuram et al., 2018), traffic occupancy (Lai et al., 2018), financial market trends (Shih et al., 2019), and aerospace data (Zhao et al., 2020). They mainly consist of sensor data sampled on multiple variables at different timestamps. In order to accurately forecast future time series, it is necessary to learn the laws and the dynamic interaction information between variables that exist in the series history (Wu et al., 2020; Xu et al., 2020; Cheng et al., 2020).
  For illustrating the dependencies in the multivariate time series, we focus on examples through case study. In Figure 1(a), a certain variable itself has periodic laws, and the dependency relationship in this respect can be well captured by the short-term and long-term model of the modeling sequence. As shown in Figure 1(b), there are dynamic dependency relations between the variables. The “arrow” indicates the trend of data changes. In 2011, the “Switzerland” variable changed significantly earlier than the other in 2013, the trends of the three variables occurred almost simultaneously.
  In literature, there have been some approaches to capture interactions in series history from different perspectives. research utilizing traditional deep learning components such as CNNs, RNNs, and attention mechanisms are more inclined to learn long-term and short-term sequence states, and fail to make full use of the entanglement between variables in the latent space (Lai et al., 2018; Shih et al., 2019; Cheng et al., 2020). New mathematical tools such as graph neural networks and tensor calculation methods have inspired the research on the spatial Relations in multivariate time series. (Wu et al., 2020) uses graph neural network to extract one-way relationships between variable pairs in the end-to-end framework. The TLASM (Xu et al., 2020) adopts tensor calculation to model the trend in the multivariate time series, and combines the core tensor components into the LSTM.
  Nevertheless, most of the above-mentioned schemes for explicitly modeling spatio-temporal dependencies tend to bulid low-dimensional variable spaces, but this is still incomplete for the variable spaces shown in Figure 1(c) of. In other words, in the high-dimensional variable space evolved over time, the ideal state for multivariate time series is that the model can capture the effective relationship between variables as fully as possible. To address the above challenges, we first define
  the high-dimensional variable space mathematically through the tensor product operator. Then, the proposed MVSRTN model models the variable space of multivariate time series.
  In the proposed scheme, N-Order Residual Tensor Network is the core component. The tensor network is based on the idea of low-rank approximation, which refers to the mode of decomposing and truncating the extremely large dependent space through tensor decomposition methods (Ma et al., 2019; Zhang et al., 2019) in the case of a certain error. For modeling the exponential variable space, we first adopt the Tensor-Train (Oseledets, 2011) algorithm satisfying the demands of processing different timestamps to decompose the variable space representation, and then obtain the space-approximated tensor network by tensor contraction. However, for long-term sequences, the continuous multiplication in the tensor contraction calculation would result in series of gradient problems. Inspired by ordinary differential equations, we extend and introduce N-order Euler discretization method, and the first-order approach can be thought of as residual connection. Finally, the space-approximated tensor network and the proposed N-order residual connection are coupled to model the effective interdependence in the variable space.
  In addition, Series-Variable Encoder aims at improving the quality of the variable space by reintegrating the time series. In this component, the combination of 1D CNN and self-attention mechanism at series level and variable level effectively enrich the dependency information mapped to the latent space. For making MVSRTN more robust, we use the linear Skip-Connection Layer to directly propagate the input scale and coding scale information to the Output Layer. Our contributions are of three-folds:
  • We represent the high-dimensional variable space, and through theoretical analysis, we demonstrate the space-approximated tensor network has the ability to model the latent variable space.
  • We propose the MVSRTN. In this architecture, a designed encoder combines long-term and short-term interactive information to optimize the variable space, a tensor network coupled by N-order residual connection captures the dynamically changing dependencies, and an additional layer propagates the scale features.
  • We perform extensive experiments on four benchmark datasets and the results demonstrate the effectiveness of MVSRTN in evaluation metrics.
sections.heading: 2 RELATED WORK
sections.text: Multivariate Time Series In the real world, a large number of multivariate time series tasks are derived from the fields of finance, economy, and transportation. The purpose of this type of task is to capture or predict the trend of a set of variables that change dynamically over time. In the research history of multivariate time series, researchers have reached a basic consensus that multiple variables are interdependent. After a period of development, statistical machine learning models such as Vector Auto-Regressive (Box et al., 2015), Gaussian Process (Frigola, 2015) algorithm can linearly model multivariate time series data, but the complexity of the model expands dramatically with the increase of time or variables. In recent years, with the rise of deep learning, neural networks implicitly capture the dependence of data in a non-linear mode. LSTNet (Lai et al., 2018) uses CNN and RNN to extract the local-term dependent mode of the time series, and extracts the long-term
  mode by recurrent-skip layer. Based on the attention mechanism and the LSTNet model, the TPALSTM (Shih et al., 2019) model can capture long-term dependencies across multiple time steps. However, algorithms based on neural networks only explicitly model the interaction in the time dimension, and fail to fully analyze the relationships in the latent variable space. The MTCNN (Wu et al., 2020) model based on graph neural network technology explicitly extracts the relationship between variables and achieves the effect of SOTA. However, only the one-way relationship between variable pairs is processed, which is not enough for the dynamic and complex variable space. Therefore, we consider using the idea of approximation to model the variable space.
  Tensor Network for Dependent Space The tensor network is a powerful mathematical modeling framework originated from many-body physics, which learns the effective associations in the exponential Hilbert space through the idea of low-rank approximation (Stoudenmire & Schwab, 2016). Based on the training approach of deep learning, Tensor networks just need to approximate the dependent space by training the decomposed parameter space. In the field of natural language, the TSLM (Zhang et al., 2019) model proves that the theoretical model obtained by tensor decomposition of the high-dimensional semantic space has a stronger ability to capture semantic dependence than the RNNs network. The u-MPS (Zauner-Stauber et al., 2018) tensor network based on Tensor-Train decomposition (Oseledets, 2011) is also effective for probabilistic sequence modeling. Tensor networks have natural advantages in the field of modeling high-dimensional dependent spaces (Miller et al., 2021). However, due to the gradient problem, the existing tensor network modeling schemes are still difficult to deal with real-world sequence data with long periods.
  Euler Discretization and Residual Connection The residual connection method can solve the problem of the gradient disappearance of the deep network (He et al., 2016). When the gradient is saturated, it can maintain the stability of the network according to the principle of identity mapping. Based on the theory of differential dynamics, the residual connection can be considered as a discretization application of the first-order Euler solver (Chen et al., 2018). Higher-order solvers (for example, the second-order Runge-Kutta) reduce the approximation error caused by the identity mapping (Zhu et al., 2019; Li et al., 2021), and also have better ability to process continuously sampled data (Demeester, 2020).
sections.heading: 3 PROBLEM STATEMENT
sections.text: In this paper, we denote the multivariate time series task as follows. Formally, given the time series signal of T points of time X = {y1,y2, · · · ,yT }>, where yt = {vt,1, vt,2, · · · , vt,D}> ∈ RD, T is the number of time steps andD is the number of variables. We leverage X ∈ RT×D as time series history to forecast the series of next time stamp yT+h, where h ≥ 0 and it is the desirable horizon which is ahead of the current time stamp. It is worth mentioning that the range of the prediction task h varies on different tasks.
  The main purpose of this paper is to model the effective variable dependency in latent space. Hence, we formulate the variable space as shown in Definition 1. Definition 1. The tensor product operator is adopted to fully interact with the variable series yt indexed by different timestamps. Taking y1 and y2 as an example, the interactive function is:
  Ty1,y2 = y1 ⊗ y2 =  v1,1v2,1 · · · v1,Dv2,1... . . . ... v1,1v2,D · · · v1,Dv2,D  (1) where Ty1,y2 ∈ RD×D, and we define the whole variable space T ∈ RD T as:
  Ty1,··· ,yT = y1 ⊗ y2 ⊗ · · · ⊗ yT (2)
  In the model architecture, we will detail how to model the proposed variable space.
sections.heading: 4 MODEL ARCHITECTURE
sections.text: In this section, we introduce the proposed MVSRTN architecture as shown in Figure 2. The SeriesVariable Encoder is utilized to optimize the original series data. The N-Order Residual Tensor
  Network approximates the variable space, captures the dynamic dependence between variables over time, and ensures the ability to model long-term sequences. We leverage Skip-Connection Layer components to memory coding and scale information. Finally, the loss function exploited by MVSRTN is mentioned.
sections.heading: 4.1 SERIES-VARIABLE ENCODER
sections.text: The Series-variable Encoder is the first part of MVSRTN, in this component, 1D CNN layer is first exploited to extract short-term patterns in the time dimension as well as local dependencies between variables:
  ct = GeLU(WC ∗ yt + bC) (3)
  where ∗ represents convolution operation, WC ∈ RkD×d and bC ∈ Rd are all the weight parameters, d is the number of filters, k is the kernel size, and GeLU is an activation function (Hendrycks & Gimpel, 2016). After the above operations and zero-padding, the new sequence representation C = {c1, · · · , cT }> ∈ RT×d can be obtained. In addition, the proposed model gains the representation that enriches the global interaction information by self-attention mechanism at series level and variable level, denoted as G ∈ RT×d,
  GS = Softmax(Tril( QSK > S√
  d ))VS
  GV = Softmax( QV K > V√
  T )VV
  G = ReLU(Concat(GS ;G > V )WG + bG)
  (4)
  where GS is the output of series-level self-attention, QS = CW Q S , KS = CW K S , VS = CW V S , and WS ∈ Rd×d are all weight parameters. GV is from variable-level self-attention, QV = C>WQV , KV = C >WKV , VV = C >W VV , and WV ∈ RT×T . Moreover, ReLU is an activation function (Nair & Hinton, 2010), Softmax means normalization, Tril refer to lower triangular matrix and WG ∈ R2d×d. A gate structure is leveraged to fuse information :
  X ′ = Sigmoid(α)×C + Sigmoid(β)×G (5)
  where α and β are all scale parameters, and Sigmoid is an function (Leshno et al., 1993) commonly used as gates. Finally, we achieve the encoded high-quality multivariate time series, and then map it to the latent variable space.
sections.heading: 4.2 N-ORDER RESIDUAL TENSOR NETWORK
sections.text: In this section, we first give a theoretical analysis of the approximate variable space of the tensor network, and secondly, introduce the algorithm flow of the component.
  In the theoretical analysis part, we build the encoded series X ′ = {x1, · · · ,xT }> ∈ RT×d into a variable space. Compared to the Definition 1, based on the time series history X ′, we represent the final variable space as follows:
  Tx1,··· ,xT = x1 ⊗ x2 ⊗ · · · ⊗ xT (6)
  In order to learn the effective interactive information in the variable space T , we decompose its representation into a tensor networkA that is contracted from T parameter tensorsW(t) ∈ Rrt−1×rt×d:
  A =W(1) · · · W(T )
  = r1∑ h1=1 r2∑ h2=1 · · · rT−1∑ hT−1=1 Wh0h1x1 W h1h2 x2 · · ·W hT−1hT xT
  (7)
  where xt ∈ Rd and ht ∈ Rrt refer to the indexes ofW(t), and the dimension of the index ht is the truncated rank of the tensor. This set of values [rt] determines the degree to which A approximates the variable space T . is tensor contraction, which can be understood as the dot product between tensors. As shown in Eq.(7), tensor contraction is essentially an addition operation on indexes with the same dimensions to eliminate this index.
  It is worth noting thatA and T are equal when the error is allowed. The above theory is demonstrated in Claim 1, which illustrates that the decomposed parameter space A has the ability to model the variable space. Claim 1. The tensor network A is obtained by tensor-train decomposition of the variable space T . When the rank of the parameter tensors in Eq.(7) is not truncated, A=T , and their complexity are all equal to O(dT ) ; When the following inequalities are satisfied, for the prescribed accuracy , A can be considered as the optimal approximation of the space T .
  ‖T − A‖F ≤ ‖T ‖F (8)
  The above is the theoretical discussion of modeling variable space through the idea of low-rank approximation. From a practical point of view, updating the tensor network through automatic differentiation is an effective way to reduce its loss with the variable space. Next, we give the specific modeling process of the N-Order Residual Tensor Network component.
  For ensuring the probability normalization of the tensor network input (Stoudenmire & Schwab, 2016), L2 regularization is performed on the sequence vector xt in X ′ at first:
  φ(xt) = L2(xt) (9)
  As shown in Figure 3(a), the normalized series φ(xt) is condensed with the network to forecast the future distribution of variables. For avoiding confusion in the position of different timestamps, the network is modeled step by step in the direction of time, and the parameter tensors
  are shared (Zauner-Stauber et al., 2018), namelyW(1) = · · · =W(T ) =W ∈ Rr×r×d:
  ht = A(φ(xt),ht−1;W) = ht W φ(xi), t ∈ [1, · · · , T ]
  (10)
  where ht ∈ Rr, h0 refers to all one vector, refers to tensor contraction and r is set as a hyperparameter bond-dimension (which is adopted to control the ability to approximate the variable space of the tensor network, and the upper limit is d T 2 ).
  For long-term sequences, only the space-approximated tensor network will result in gradient problems. In response to the issue, we propose the N-order residual connection approach. First, compared with the tensor network structure mentioned above, we add the residual term when modeling the series information of the current time stamp (as shown in Figure 3(b)):
  ht = H(φ(xt),ht−1;W) = ht−1 + A(φ(xt),ht−1;W), t ∈ [1, . . . , T ]
  (11)
  Inspired by the numerical solution of ordinary differential equations, its high-order solvers have lower truncation error and higher convergence stability, we propose a tensor network combined with the N-order residual connection as shown in Figure 3(c):
  ht = H(φ(xt),ht−1;W)
  = ht−1 + N∑ j=1 ajzj , t ∈ [1, . . . , T ] (12)
  { zj = A(φ(xt),ht−1;W) j = 1 zj = A(φ(xt) + bj ,xt−1 + bjzj−1;W) 2 ≤ j ≤ N
  where aj and bj are all fixed coefficients. Please see the Appendix A for the coefficient values calculated by the Taylor’s Formula of the different order residual connections.
  xU = hTWU + bU (13)
  Finally, a linear component maps hT ∈ Rr to xU ∈ RD, where WU ∈ Rr×D and bU are all parameter weights.
sections.heading: 4.3 SKIP CONNECTION LAYER
sections.text: The layer is added after the input layer and encoder layer respectively, which directly connected to the Output Layer. For multivariate time series, numerous nonlinear structures in the model make the output scale insensitive to the input scale (Lai et al., 2018), resulting in a significant reduction in the prediction accuracy of the model. In addition, in order to memorize the encoder states, we also adopt the layer to directly propagate the encoding information to the output.
  xL = LWL + bL (14)
  the known input data is X = {y1,y2, · · · ,yT }> ∈ RT×D, and the selected series is L = {yT−l+1, · · · ,yT } ∈ RD×l, WL ∈ Rl×1 and bL ∈ R1 are all weight parameters.
  xM = (MWM + bM )WO + bO (15)
  the selected series is M = {xT−m+1, · · · ,xT } ∈ Rd×m, where the encoder series is X ′ = {x1,x2, · · · ,xT }> ∈ RT×d, and weight parameters are WM ∈ Rm×1, bM ∈ R1, WO ∈ Rd×D and bO ∈ RD.
sections.heading: 4.4 OUTPUT LAYER AND LOSS FUNCTION
sections.text: Finally, the prediction result at time stamp s = T + h of the network is Y ′s ∈ RD:
  Y ′s = x U + xL + xM (16)
  Moreover, we utilize the validation set to decide which of the two loss functions is better for our model. we leverage Mean Absolute Error (MAE, also denoted as L1 loss function) and Mean Square
  Error (MSE, also denoted as L2 loss function) to update the network parameters. MSE is simple to calculate, which is the default loss function of many prediction tasks, and MAE has better robustness for outliers.
  MSE = ∑
  s∈Ωtrain D∑ i=1 (Ys,i − Y ′s,i)2
  MAE = ∑
  s∈Ωtrain D∑ i=1 |Ys,i − Y ′s,i|
  (17)
  where Ωtrain is a set of time stamps for model training.
sections.heading: 5 EXPERIMENTS
sections.text: In the section, we perform extensive experiments on four multivariate time series forecasting benchmark datasets between MVSRTN and seven baselines. Moreover, in order to evaluate the effectiveness of the proposed model, we conduct ablation experiments and order experiments.
sections.heading: 5.1 DATASETS AND BASELINES
sections.text: Our experiments are based on three datasets which are publicly available:
  • Traffic 1: This dataset is composed of 48 months hourly data from the California Department of Transportation during 2015 and 2016. The data describes the road occupancy rates (between 0 and 1) measured by different sensors on San Francisco Bay area freeways.
  • Exchange-Rate: A dataset that consist of the daily exchange rates of eight countries including Australia, British, Canada, Switzerland, China, Japan, New Zealand and Singapore ranging from 1990 to 2016.
  • Solar-Energy 2: This dataset consists of the solar power production records in the year of 2006 and these records are sampled every 10 minutes from 137 PV plants in Alabama State.
  • Electricity 3: This dataset from the UCI Machine Learning Repository is composed of the electricity consumption in kWh which was recorded every 15 minutes from 2012 to 2014 for 321 clients.
  We compare with a number of forecasting methods. AR: An auto-regressive model for time series forecasting. GP (Frigola, 2015; Roberts et al., 2012): A Gaussian Process time series model for prediction. LSTNet-skip (Lai et al., 2018): A LSTNet model (A deep neural network that combines convolutional neural networks and recurrent neural networks, and could catch the long-term and short-term patterns of time series) with skip-RNN layer. LSTNet-Attn (Lai et al., 2018): A LSTNet model with temporal attention layer. TPA-LSTM (Shih et al., 2019): An attention-recurrent neural network for multivariate time series forecasting. MTGNN (Wu et al., 2020): A graph neural network model for the forecasting of multivariate time series. MTGNN-sampling (Wu et al., 2020): MTGNN model trained on a sampled subset of a graph in each iteration.
  All methods are evaluated with two metrics: root relative squared (RSE) and empirical correlation Coefficient (CORR). Lower value is better for RSE while higher value is better for CORR. In addition, the details of the datasets and metrics are in Appendix B.
sections.heading: 5.2 TRAINING DETAILS
sections.text: For all tasks, we implement our model with Pytorch-1.20, and train them on a NVIDIA GeForce RTX 2080 Ti GPU. For Traffic and Solar datasets, the size of the series history window T is 168, for Solar-Energy it is 144, and for Exchange-rate it is 120. All the weight parameters are initialized with Xavier (Glorot & Bengio, 2010) and normalized by weight normalization (Salimans &
  1http://pems.dot.ca.gov 2http://www.nrel.gov/grid/solar-power-data.html 3https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
  Kingma, 2016). As for learning method, we use the Adam optimizer (Kingma & Ba, 2014) and an exponentially decaying learning rate with a linear warm up. The initial learning rate is set from 0.0005 to 0.001 and the batch size is 128. We perform dropout after each layer, except input and output ones, and the rate usually is set to 0.1 or 0.2. The 1D CNN component has 50 filters, the filter size is set from 3 to 6. In the tensor network, the bond-dimension r is tuned range from 50 to 100, and the order of the residual connection is chosen from {1,2,4}. For the chosen size l and m in Skip-Connection Layer, the l is 24 for all datasets. and the m is set to the history window size T for different datasets.
sections.heading: 5.3 MAIN RESULTS
sections.text: Table 1 shows the prediction performances of all the models on all datasets, and the metrics are RSE and CORR. We set horizon = {6, 12}, respectively, which means the horizons are set to 6 or 24 days over the Exchange-Rate dataset, to 6 or 24 hours over the Electricity and Traffic task, to 60 or 240 minutes over the Solar-Energy data. It is worth noting that the larger the horizons, the harder the prediction tasks. The best result for each metric of different datasets is highlighted in bold face in the table.
  Compared with the MTGNN model (state-of-the-art method), our proposed model achieves competitive results. In particular, for the exchange-rate dataset, the results of MVSRTN on the four (horizon, metric) pairs all outperform the MTGNN model. For the Solar-Energy data when horizon=24, MVSRTN lowers down RSE by 2.46%. In general, in addition to the traffic dataset, our method has a greater advantage when the horizon is 24. The reasons are that the graph modeling method of MTGNN is more suitable for traffic data, and when the series data is difficult to predict (horizon=24), our model plays the role of modeling semantic space.
sections.heading: 5.4 ABLATION EXPERIMENTS
sections.text: To demonstrate the effectiveness of each model structure on Solar-Energy, Traffic, Electricity and Exchange-Rate data, we compare MVSRTN with 4 variants as follows:
  • MVSRTN-nA: We remove the self-attention at series and variable level component such that there is a lack of global interactive information at series level and variable level.
  As shown in Figure 4, reported results are the average of 10 runs. For all the datasets, removing the residual tensor networks and removing the skip-connection layer both cause significant performance degradation. In particular, for the CORR metric of the Exchange-Rate dataset, the MVSRTN-nS variant basically loses the ability to accurately forecast, which means the exchange rate data is more sensitive to scale information. Moreover, solar energy and electricity data are more obviously affected by residual tensor network components.
sections.heading: 5.5 ORDER ANALYSIS
sections.text: In order to verify the effects of different orders of the residual tensor network, as shown in Figure 5, we design the comparative experiments over the orders of 1, 2, 4 on the Solar-Energy, Traffic, Electricity and Exchange-Rate datasets. The experimental results show that the second-order residual tensor network (RTN) achieves the best results on both RSE and CORR metrics for solar energy data and exchange rate data. For traffic data, the effect of fourth-order RTN is better than that of first-order and second-order. In addition, the result of the fourth-order RTN is similar to the secondorder RTN on the Electricity dataset. This verifies the effectiveness of the proposed higher-order residual connection.
sections.heading: 6 CONCLUSION
sections.text: In this paper, we propose a novel framework (MVSRTN) for fully exploiting the dependencies in the variable space in the task of multivariate time series forecasting. We exploit a tensor network based on the idea of low-rank approximation to model the variable space and shared tensor components to ensure the translation invariance of the network. Additionly, we propose an N-order residual connection approach and couple it to the space-approximated tensor network to improve the ability to model long-term sequences. Moreover, a series-variable encoder is designed to improve the quality of the variable space, and we leverage the skip-connection layer to achieve the dissemination of information such as scale. According to the comparison with 7 baselines, we show the efficiency of the framework of MVSRTN , and experimental results show the effect of modeling variable space for multivariate time series forecasting.
sections.heading: A THE COEFFICIENT VALUES OF RESIDUAL TENSOR NETWORK
sections.text: Table 2 shows the coefficients used in Equation 4.2.
sections.heading: B DATASETS AND METRICS
sections.text: In Table 3, we summarize statistics of benchmark datasets. Moreover, two conventional evaluation metrics exploited by us are defined as follows:
  Root Relative Squared Error (RSE):
  RSE =
  √∑ s∈Ωtest(Ys,i − Y ′ s,i)
  2√∑ s∈Ωtest(Ys,i −mean(Y )) 2 (18)
  Empirical Correlation Coefficient (CORR):
  CORR = 1
  n n∑ i=1 ∑ s(Ys,i −mean(Yi))(Y ′s,i −mean(Y ′i ))√∑ s(Ys,i −mean(Yi))2(Y ′s,i −mean(Y ′i ))2
  (19)
  where Ωtest is a set of time stamps for model test, and lower value is better for RSE while higher value is better for CORR.