Real-world dynamical systems often consist of multiple stochastic subsystems that interact with each other. Modeling and forecasting the behavior of such dynamics are generally not easy, due to the inherent hardness in understanding the complicated interactions and evolutions of their constituents. This paper introduces the relational state-space model (R-SSM), a sequential hierarchical latent variable model that makes use of graph neural networks (GNNs) to simulate the joint state transitions of multiple correlated objects. By letting GNNs cooperate with SSM, R-SSM provides a flexible way to incorporate relational information into the modeling of multi-object dynamics. We further suggest augmenting the model with normalizing flows instantiated for vertex-indexed random variables and propose two auxiliary contrastive objectives to facilitate the learning. The utility of R-SSM is empirically evaluated on synthetic and real time series datasets.
Summary of what the paper claims and contributes
---
This paper proposes a hierarchical latent variable model of sequential dynamic processes of multiple objects when each object exhibit significant stochasticity. The model leverages Graph Neural Networks to design the architecture of the model. Normalizing flows are used to construct "Graph Normalizing Flows" that model transition densities of the per-object latent random variables, although the full model has more latent variables than tabouthe per-object latents. The approach applies and combines a host of methods to design the objective, the model, and optimize the model under the objective -- for the objective: variational inference, Contrastive Predictive Coding; for the model: normalizing flows, graph neural networks (and their combination, termed by the authors as Graph Normalizing Flows); for the optimization: variational sequential Monte Carlo. From my understanding, the main claims are that the model is a flexible way to incorporate relational information, and that the resulting model outperforms other multi-object time-series forecasting approaches.
Experiments are conducted on a synthetic toy dataset of 1d observations per-object, a real dataset of basketball player movement, corresponding to 2d observations of each object's position (the agent's and the ball's positions), and a real dataset of traffic speed forecasting (1d observations per-object). The experiments demonstrate that the model compares favorably to a baseline and a few other time-series forecasting models from the literature. The experiments show that the method achieves higher test log-likelihood (lower-bounds), and lower MSE than the compared methods. The paper discussed many related works, but it's not clear why the specific methods were chosen for comparisons. More motivation is needed for these comparisons.
Evaluation
---
>Originality:
Are the tasks or methods new?
The method is new.
Is the work a novel combination of well-known techniques?
Yes, albeit a lot of techniques.
Is it clear how this work differs from previous contributions?
Besides the fact that the proposed method is a combination of many techniques, the main / key technical difference was actually unclear to me.
Is related work adequately cited?
From my understanding, yes, mostly; there's just a few places where the paper could be situated better:
- Sec 1: "as they have been shown to be fundamental NN building blocks ...". Citations needed (e.g. include the Graph NN citations later here)
- Sec 1: "R-SSM achieves superior test likelihood and good prediction performance". When compared to what methods (citations to VRNN and GNN-AR/NRI needed here too)?
- The paper contrasted the work to other dynamics modeling works with the statement "R-SSM differs from all these works by introducing structured latent variables to represent the uncertainty on state transition and estimation". The paper also said VRNNs were the most related, because they employ "hidden states of per-agent VRNNs interact through GNNs". The paper (PRECOG, arXiv:1905.01296) appears related as it also constructs a latent variable graphical model of stochastically-transitioning multiple interacting objects with normalizing flows. Like the authors said of Graph VRNNs, it appears that one of the biggest differences to PRECOG looks like the proposed method includes more sources of stochasticity outside of the objects (the global uncertainty) to model the observations.
>Quality:
Is the submission technically sound?
Yes, although not much motivation is given for many of the specific design decisions adopted. The model is a complicated combination of many techniques from the literature, which makes it difficult to understand the importance of each specific piece of the model.
- For instance, why was global stochasticity introduced? Because per-object stochasticity is modeled with normalizing flows, the model would have been trainable with maximum likelihood estimation of a distribution over observations that's efficiently analytically computable, but after the addition of global stochasticity, the joint distribution over the random variables becomes intractable, and training must resort to variational inference. This leads to, as the paper mentioned, RSSM suffering from posterior collapse, which, to my knowledge, does not occur when training direct maximum likelihood estimation procedures (e.g. GLOW, RealNVP).
- As mentioned above, it's not clear what the key technical innovation of this approach is (Graph Normalizing Flows?) It would be good if the claimed innovation was made clear in both the abstract and the introduction.
Are claims well supported by theoretical analysis or experimental results?
The experimental results provide evidence that 1) the model can flexibly incorporate relation information and some evidence that 2) the model outperforms some multi-object time-series forecasting approaches from the literature.
Is this a complete piece of work or work in progress?
The submission appears complete.
Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?
Yes, and they included discussion of when the model is under-performing w.r.t. other methods (at longer time horizons).
>Clarity:
Is the submission clearly written?
As mentioned above, many of the numerous design decisions are not motivated well. Essentially, the paper is not very self-contained.
Is it well organized?
Yes.
Does it adequately inform the reader?
No -- the paper assumes a lot of prior knowledge, which most potential readers would likely possess. This issue would be less significant if the main innovation was clearer.
>Significance:
Are the results important?
It's difficult to tell, because it's not very clear why, of the many cited works, the specific ones chosen for comparison were used. More motivation for the comparison is needed.
Are others (researchers or practitioners) likely to use the ideas or build on them?
Possibly, although the model is complicated, which makes it less likely.
Does the submission address a difficult task in a better way than previous work?
It's plausible that the model performs better, but it's unclear how widely-applicable the method is. The model appears to be restricted to lower dimensional settings.
Does it advance the state of the art in a demonstrable way?
It's plausible, but I am not sure, as mentioned above.
Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?
No
Additional feedback
---
Sec 5.2 describe what "OOB rate of the rollouts" means.
Sec 3.3 Coupling layer equation is missing a final parenthesis.
3.3 "complicate the learning a lot." -> "significantly complicate the learning." (original is subjectively too informal)
Sec 5.2 "like to dig into it in future work" -> "like to explore it in future work" (original is subjectively too informal)
Recommendation
---
It is unclear what the main key contributions are, outside of usually performing better than some related methods. This lack of clarity, along with a complicated design that is not very well-motivated, make me lean towards rejection. An improved version of this paper would make clear what the precise contributions are, explain how the related work did not achieve them, and why the specific comparisons were adopted.