Deep learning based approaches have been widely used in various urban spatiotemporal forecasting problems, but most of them fail to account for the unsmoothness issue of urban data in their architecture design, which significantly deteriorates their prediction performance. The aim of this paper is to develop a novel clustered graph transformer framework that integrates both graph attention network and transformer under an encoder-decoder architecture to address such unsmoothness issue. Specifically, we propose two novel structural components to refine the architectures of those existing deep learning models. In spatial domain, we propose a gradient-based clustering method to distribute different feature extractors to regions in different contexts. In temporal domain, we propose to use multi-view position encoding to address the periodicity and closeness of urban time series data. Experiments on real datasets obtained from a ride-hailing business show that our method can achieve 10%-25% improvement than many stateof-the-art baselines.
Summary: This paper proposes a clustering attention-based approach to handle the problem of unsmoothness while modeling spatio-temporal data, which may be divided into several regions with unsmooth boundaries. With the help of a graph attention mechanism between vertices (which correspond to different regions), the CGT model is able to model the (originally unsmooth) cross-region interactions just like how Transformers are applied in NLP tasks (where words are discrete). Experiments seem to suggest a big improvement when compared to baselines.
Pros:
+This should be one of the first works that apply a graph transformer alike method in this domain, and specifically on the unsmoothness problem.
+ Since the dataset is not publically available, there aren't many prior works to compare the CGT to. However, at least compared to the one prior work [1] that the authors point to in Section 4, the RMSE results achieved CGT does seem to be significantly better.
========================================
However, I still have some questions/concerns on the paper, detailed below.
1) The current organization of the paper, as well as its clarity, can (and should) be significantly improved. I didn't completely understand the approach on my first two passes, and I **had** to read the code published by the authors. Here are some issues that I found:
  - For one, Figure 2 is not quite helpful as it's too messy with font size too small. A similar problem is with Figure 4 which, without further clarification (e.g., of what "atrous aggregation" exactly mean), is very hard to interpret. 
  - The notations are very inconsistent and messy:
      i) In Eq. (1), you should use a symbol different from $\mathbf{X}$ to refer to the "predictions". Since you are applying $f(\cdot)$ on $\mathbf{X}_{t-T_x+1:t}$, you should not get the "exact same" target sequence. That's your target. Maybe use $\hat{\mathbf{y}}$, which you used in Eq. (8).
      ii) In Figure 3, what is the orange line? In addition, I only saw two blue lines in the figure, but the legend seems to suggest there are four of them...
      iii) The notations used in Figure 4 are somewhat confusing. For example, what does "f->1" mean? (I later found through Eq. (2) that it means transform to 1 dimension; but the small plots in Figure 4 suggest f is a "magnitude" of the feature.) In addition, there are two $H_1$ in Figure 4 with clearly different definitions.
      iv) The authors used $\mathcal{G}_{\theta_k}(x_i)$ in Eq. (3) without defining it. The definition actually came much later in the text in Eq. (6). I suggest moving the usage of the clustering assignment (i.e., Eq. (3)) to after Eq. (6).
      v) What does $[\cdot || \cdot]$ mean (cf. Eq. (4))? (The code seems to suggest it's concatenation?)
      vi) The authors first used $h_{x_i}$ in Eq. (3) to denote the output of the CAB module. Then letter $h$ is then re-used in Eq. (4) and (5) with completely different meanings. For instance, the $W_kh_i$ in Eq. (4) correspond to line 48 of the code "model.py". (By the way, nowhere around Eq. (4) did the authors explain how $h_i$ is produced, such as taking the mean over the batch dimension, etc.). 
      vii) In Section 2.6, you denote the "optimal vertex cluster scheme" with letter $C$, which is used in Eq. (2). Similar for parameter $a_k$ and atrous offset $a$.
  
  - This not a very big problem (as it seems somewhat inevitable), but I think there are too many acronyms in the paper.
  
  I think it'd be great if the authors can take care of these issues, as clarity in math and descriptions are critical to the presentation of such an involuted method. It would also be useful to clearly define the dimensionality of all the variables (e.g., you defined $V$ in Section 2.1, but never used it again in later subsections).
2) Regarding the usage of the multi-view position encoding, the authors claimed that it "provides unique identifiers for all time-steps in temporal sequences". However, if you consider $x=7$ and $x=14$, then $PE_i(7)=PE_i(14)$ for all $i=1,2,3,4$ with $PE_5(7) \approx PE_5(14)$. Doesn't this invalidate the authors' claim? Also, doesn't this mean that the proposed MVPE only works on sequences with length <= 7? (In comparison, the design of positional encoding in the original Transformer doesn't have this problem.)
(You didn't show how you implemented and initialized the position encoding in the uploaded code, so I may be missing some assumptions here.)
3) In line 48 of the code (https://github.com/CGT-ICLR2020/CGT-ICLR2020/blob/master/model.py#L48), why did you take the mean over the batch dimension? Shouldn't different samples in a minibatch be very different? Does a (potentially completely independent) sample in a batch affect another sample? A similar problem occurs for Eq. (9): Why do you require clusterings of two different samples $b_1, b_2$ to be similar? (Where these samples can come from quite different times and years of the data?) 
4) In the experiments, you "sampled 10 input time-steps" due to computational resources. Typically, in Transformer-based NLP tasks the sequence lengths can be over 500, with much higher dimensionality (e.g., 512); but you are only using sequence length 10 and dimensions <= 16 (in your code, you used "self.dec_io_list = [[5,8,16,16],[16,16,16,16],[16,8,8,8]]"). What is the bottleneck for the computation of your approach? (I noticed there are more than 1K vertices in city A, which may be a costly factor indeed.) How much memory/compute does the CGT method consume? How does using a longer sequence affect the performance of CGT?
5) You performed an ablation study on MVPE. Did you simply remove MVPE, or did you use the conventional PE from the original Transformer paper (Vaswani et al. 2017)? (If the latter, I'm very surprised that MVPE is so much better than PE. In that case, you may want to try MVPE on NLP tasks to see if it also improves SOTA.)
6) How did you measure unsmoothness in Figure 6? It doesn't seem like a quantifiable property to me. You should discuss this in the experiment section.
-----------------------------------
Minor questions/issues that did not affect the score:
7) There are some strange phrases/sentences in the paper. For example, the first sentence of the 2nd paragraph of Section 1: "we will show **throughout the paper** that urban spatiotemporal prediction task suffers from..."
8) Why use an encoder-decoder architecture at all? Why can't we train the model like in language modeling tasks, where we want to predict the next token? In other words, you can simply use a decoder-side CGT, and mask the temporal self-attention as in the Transformers.
-----------------------------------
In general, I think this paper proposed a valuable approach that seems to work very well on the spatio-temporal dataset they used (which unfortunately is private). However, as I pointed out above, I still have numerous issues with the paper's organization and clarity, as well as some doubts over the methodology and the experiment. I'm happy to consider adjusting my score if the authors can resolve my concerns satisfactorily.
[1] http://www-scf.usc.edu/~yaguang/papers/aaai19_multi_graph_convolution.pdf