The Hamiltonian formalism plays a central role in classical and quantum physics. Hamiltonians are the main tool for modelling the continuous time evolution of systems with conserved quantities, and they come equipped with many useful properties, like time reversibility and smooth interpolation in time. These properties are important for many machine learning problems -from sequence prediction to reinforcement learning and density modelling -but are not typically provided out of the box by standard tools such as recurrent neural networks. In this paper, we introduce the Hamiltonian Generative Network (HGN), the first approach capable of consistently learning Hamiltonian dynamics from high-dimensional observations (such as images) without restrictive domain assumptions. Once trained, we can use HGN to sample new trajectories, perform rollouts both forward and backward in time and even speed up or slow down the learned dynamics.
1 We demonstrate how a simple modification of the network architecture turns HGN into a powerful normalising flow model, called Neural Hamiltonian Flow (NHF), that uses Hamiltonian dynamics to model expressive densities. We hope that our work serves as a first practical demonstration of the value that the Hamiltonian formalism can bring to deep learning.
The paper proposes two ideas: 1) Hamiltonian Generative Networks (HGN) and 2) Neural Hamiltonian Flow (NHF). 
Hamiltonian Generative Networks are generative models of high-dimensional timeseries which use hamiltonian differential equations to evolve latent variables (~position and ~momentum vectors) through time (using any differentiable integration scheme). Given the ~position vector a decoder network generate predictions. The initial latent variables are inferred using a VAE style inference network that takes a sequence of images as the input. The decoder, inference network and crucially the hamiltonian energy function is learned by minimizing a VAE style ELBO on observed sequences. The model induces a strong hamiltonian physics prior and is quite elegant all in all. The model is evaluated on 4 simulated physics tasks, and beats the only baseline the Hamiltonian Neural Network (HNN).
Neural Hamiltonian Flow notes that hamiltonian dynamics are invertible and volume preserving, which is the properties you need for neural flow models. As such it propose to use a series of hamiltonian update steps with multiple learned energy functions as a flexible density estimator. The resulting density estimator is subjectively evaluated on three 2d toy density estimation tasks.
I propose a weak accept as I think the paper is interesting and well written, but could be much better. The paper explains how both HGN and NHF work, but not much more. The HGN is only compared to a single other method (the closely related HNN), on four toy benchmarks. The NHF is barely evaluated, and not compared to anything.
Does the authors actually care about modelling physics and think their method is superior at this? If so, they should compare and contrast to some of the many, many papers on modelling physics, e.g. [1,2,3,4] and references herein. If not, what do they care about? Where do they think this model can be useful? Why should anyone use this model over some of the many, many other models one could use?
Similarly for the NHF, if I only read this paper I have no idea whether it's better than any of the other flow based models. Is it faster (to sample? to eval likelihood?) is it a better estimator? Why should I use it?
I think the paper would benefit from being split into two papers, each thoroughly examining one idea.
A few questions and minor comments
 - While the hamiltonian dynamics expect position and momentum vectors, the neural network is free to use those however it sees fit. Actually, if I understand correctly, the position vector must also encode the color of the objects for the 2 and 3 body problem. Is that correct? It would be interesting if you could examine how predictive the q and p vectors were of the true position and momentum vectors. 
 - Successful experiments with n-body problems with n randomly sampled during training and unseen n used in testing would be very powerful in showing generalization. I'm afraid that the current setup doesn't generalize well.
 - I'm surprised that the generated images start showing artifacts after some time, e.g. pendulum sample 4 and 6 in https://docs.google.com/presentation/d/e/2PACX-1vRD2FnKgymgR2lU8lE6-XM8Cz-UWLTI6n_Uht3v6Gu4hIyMHmOcNL5D-0eG6Z4WHDAWS4qFosU-lxXP/pub?start=false&loop=false&delayms=3000&slide=id.g61bbdf339d_0_426. How can those appear if the hamiltonian dynamics preserve energy?
 - Equation 3 is given as self evident. It's not clear to me why 1) det(I+dt*A) = 1+dt*Tr(A)+O(dt^2). Can the authors give a reference? Also, doesn't the O(dt^2) term accumulate over multiple timesteps or longer rollouts? if so, how can the multiple steps proposed be said to be volume preserving? 
[1] - Battaglia, Peter, et al. "Interaction networks for learning about objects, relations and physics." Advances in neural information processing systems. 2016.
[2] - de Avila Belbute-Peres, Filipe, et al. "End-to-end differentiable physics for learning and control." Advances in Neural Information Processing Systems. 2018.
[3] - Santoro, Adam, et al. "A simple neural network module for relational reasoning." Advances in neural information processing systems. 2017.
[4] - Fraccaro, Marco, et al. "A disentangled recognition and nonlinear dynamics model for unsupervised learning." Advances in Neural Information Processing Systems. 2017.