Recently, neural-network based forward dynamics models have been proposed that attempt to learn the dynamics of physical systems in a deterministic way. While near-term motion can be predicted accurately, long-term predictions suffer from accumulating input and prediction errors which can lead to plausible but different trajectories that diverge from the ground truth. A system that predicts distributions of the future physical states for long time horizons based on its uncertainty is thus a promising solution. In this work, we introduce a novel robust Monte Carlo sampling based graph-convolutional dropout method that allows us to sample multiple plausible trajectories for an initial state given a neural-network based forward dynamics predictor. By introducing a new shape preservation loss and training our dynamics model recurrently, we stabilize long-term predictions. We show that our model's long-term forward dynamics prediction errors on complicated physical interactions of rigid and deformable objects of various shapes are significantly lower than existing strong baselines. Lastly, we demonstrate how generating multiple trajectories with our Monte Carlo dropout method can be used to train model-free reinforcement learning agents faster and to better solutions on simple manipulation tasks.
Overview:
This paper introduces a method for physical dynamics prediction, which is a version of hierarchical relation network (Mrowca ‘18). HRNs work on top of hierarchical particle-based representations of objects and the corresponding physics (e.g. forces between different parts), and are essentially are graph-convolutional neural networks.
Unlike the original work, the proposed method introduces several improvements: 1. an updated loss function, that adds distance constraints between *all* the particles of the object. 2. recurrent training, when the predictions are fed to the inputs. 3. adding dropout.  
Writing:
The paper is relatively well-written and easy to follow.
Evaluation:
Authors compare their model on a dynamics prediction task and seem to outperform the original HRN, especially on longer-term sequences. In addition they report results for trajectory sampling (qualitative) and model-free RL, where using their model as a stochastic simulator seems to have positive impact on agent training.
Decision:
Although the proposed improvements upon HRN generally make sense, it is not clear if those are very significant on their own: adding dropout and recurrent training do not seem particularly novel and, since there is no ablation study, it is hard to see what exactly contributes to the reported improvements. 
As for the experimental evaluation, it seems like important baselines are missing, and the model seems to be very sensitive to hyperparameters (see questions). Thus, I am currently leaning more towards a rejection, hence the “weak reject” rating.
Various questions / concerns:
* It is not clear why authors do not provide a comparison to DPI-Nets (Li ICLR‘19). This model seems to be outperforming HRN, and from what it looks like is publicly available: https://github.com/YunzhuLi/DPI-Net. I would encourage authors to provide comparison to this baseline, and potentially on similar sets of experiments, or explain why this comparison would not be possible (which seems unlikely).
* Authors seem to acknowledge that the model is sensitive to the hyperparameter choice (dropout rates), however, there is no numerical evaluation that would help readers understand how critical this choice is for the final performance. Judging from very specific settings in different experiments, this could be a serious concern.
* I find it a bit strange that results in Fig. 7-8 are for random seeds. Is it not possible to just plot an average for e.g. 10 runs?
Update:
I would like to thank the authors for a detailed response!
It seems like there is a common concern about the novelty among reviewers: improvements over HRN are quite incremental. Although authors provide a verbal justification for not comparing to another strong baselines, I do not see why would it not be possible to compare methods in the similar settings, even though that baseline might be more limited.
Generally, if the main contribution is actually only the "stochastic" part and not improved performance, then just adding dropout does not seem like a particularly novel approach to me: whether the convolutions are on graphs or on euclidean domains, this does not change the way dropout is done.