Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements. For example, air quality monitoring system records PM2.5, CO, etc. The resulting time-series data often has missing values due to device outages or communication errors. In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps. Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly. In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data. In order to jointly capture the self-attention across different dimensions (i.e. time, location and sensor measurements) while keep the size of attention maps reasonable, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner. On three real-world datasets, including one our newly collected NYCtraffic dataset, extensive experiments demonstrate the superiority of our approach compared to state-of-the-art methods for both imputation and forecasting tasks.
This paper proposes a Transformer-based model with cross-dimensional self-attention for multivariate time series imputation and forecasting. The authors consider time series with observations collected at different locations (L), for different measurements (M), and at different timestamps (T). The authors describe 4 different self-attention mechanisms based on how the three dimensions, and it turns out the proposed Decomposed approach achieves the best performance and has moderate model complexity among the four. Experiments on several traffic and air quality datasets show the superiority of the proposed model.
Overall the problem and the proposed model are well motivated. Handling time series data with missing values is quite important, and the authors design the novel cross-dimensional attention mechanism which is reasonable and performs well. Especially, the authors compare several recent RNN-based models. The proposed method outperforms these strong baselines in imputation and long-term forecasting tasks.
However, I do have a few questions and concerns.
It would be quite helpful to see the training time and model size comparisons with baselines, which is to validate the claim in the introduction that `replacing the conventional RNN-based models to speed up training`.
The proposed model treats all three dimensions equally, with direct attention of every two variables, and is independent with the order. Though effective and mathematically clean, I am not sure the temporal/spatial smoothness and dependencies in time series are properly modeled in this way -- as time series is not the same as embedded sequences in NLP. This may explain why the performance on short-term forecasting is relatively unsatisfying.
It seems that the proposed model is designed for missing completely at random (implied by the statement `... due to unexpected sensor damages or communication errors` from the introduction, and the experimental settings on adding missing values). Many missing variables in time series may be missing at random or even not at random.
About the experiments:
Two datasets of the three mentioned in the main paper have M=1, which degrade the proposed model from 3-dimensional to 2-dimensional.
Why forecasting experiment is conducted on METR-LA, while using the imputation for forecasting is conducted on a different dataset and without comparing other forecasting baselines?
What are the metrics used in Tables 6, 7, 9? Several metrics are used (RMSE, MSE, MAPE, MAE, MRE) while results on different datasets are shown in different but not all metrics. Is there any reason to cherry-pick metrics for different experiments?
In Table 4, the proposed method's results in RMSE is consistently better than shown in MAE compared with other baselines. Any explanations would be useful.
The overall idea is relatively easy to follow, while some detailed descriptions should be added or clarified.
When taking health-care data as an example of geo-tagged time series, could you explain or provide references?
Figure 2 only demonstrates 3 attention mechanisms, and the Shared should also be included.
S(i,j) is used in Section 3.1 without explicit formal definition.
Please clarify the sentence about \sigma below Equation (4).
Please refer to the section number explicitly in Supp if used. (E.g., on Pages 5 and 7.)
On which dataset and what settings are the results in Table 1 computed? The numbers are helpful, but it would be better if the results are computed based on the hyperparameters (e.g., T, L, M, d_V, etc).
Minor typos:
Page 3, Paragraph of RNN-based data imputation methods: `...indistinguishable. so that...`
Page 3, Section 3.1, `..where Then, ...`
Page 14, A' is used to denote the reshaped tensor, while \tilde is used in the main paper.
Page 16, `During testing,`