Recently various neural networks have been proposed for irregularly structured data such as graphs and manifolds. To our knowledge, all existing graph networks have discrete depth. Inspired by neural ordinary differential equation (NODE)   for data in the Euclidean domain, we extend the idea of continuous-depth models to graphs, and propose graph ordinary differential equation (GODE). The derivative of hidden node states are parameterized with a graph neural network, and the output states are the solution to this ordinary differential equation. Noticing that NODEs are typically trained with the adjoint method, with the advantages of adaptive evaluation and a free-form continuous invertible model; however, their performance on benchmark image classification tasks is significantly inferior to discrete-layer models. We show the reason is adjoint method generates inaccurate gradient due to numerical error in reverse-mode integration. We then propose a memory-efficient framework for free-form ODEs with accurate gradient estimation, which is fundamental to deep-learning models. Furthermore, when applied to invertible blocks, our method achieves constant memory cost with GODE (NODE). We show our method for free-form ODEs generalizes to various model structures and achieves high accuracy for both NODE and GODE in benchmark tasks.
Summary: This paper proposed a deep model called graph-ODE (GODE), which extends the continuous deep model, neural ODE [1], to graph structured data. Two methods of computing the gradient are proposed, one is the adjoint method proposed in [1] which is memory-efficient but not perfectly accurate, and the other method is to discretize the ODE and compute an accurate gradient, which is also memory-efficient but relies on an invertible architecture.
In general, this paper contains many interesting thoughts, but to me, it seems these ideas are not new and have been proposed before[1,2,3,4]. I could not find a strong contribution of this paper. 
- - - about the ODE stability issue
Among invertible deep models, the advantage of ODE-based continuous models [1] is: there is *no restriction* on the form of f. The drawback is the computed gradient has an error, depending on discretization step size and stability. This paper pointed out the stability problem (which is also discussed in [2,3]), but do not provide a solution in the domain of continuous deep models. 
Instead, the solution they provided is to use a discretized version and compute the gradient accurately. Then it becomes a standard invertible DL model with discrete layers, where the invertible building blocks have a specific *restricted form*. The 'adjoint method with discrete-time' in Eq (10) is the same as the chain rule, which has also be pointed out in [4]. To this point, I think GODE is in the class of discrete invertible DL models trained by gradient descent. I think it less related to continuous models, except the step size can be adaptive in the forward pass.
- - - about the invertible building block
The proposed invertible building block replaces 'sum' in [5] by a function \psi. This is not novel enough to serve as a contribution.
- - - comparison to graph neural network
I think it is interesting to apply ODE techniques or other invertible architectures to graph-structured data, for which I didn't see similar works before and could be a contribution of this paper. However, for the experimental results shown in table 3 and 4, the improvement is really small. A stronger result is needed to demonstrate the advantages.
[1] Chen, Tian Qi, et al. "Neural ordinary differential equations." Advances in neural information processing systems. 2018.
[2] Chang, Bo, et al. "Reversible architectures for arbitrarily deep residual neural networks." Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
[3] Behrmann, Jens, et al. "Invertible Residual Networks." International Conference on Machine Learning. 2019.
[4] Li, Qianxiao, et al. "Maximum principle based algorithms for deep learning." The Journal of Machine Learning Research 18.1 (2017): 5998-6026.
[5] Gomez, Aidan N., et al. "The reversible residual network: Backpropagation without storing activations." Advances in neural information processing systems. 2017.
-------------after reading the response
I'd like to thank the authors for their explanations.
However, the authors' explanation of the benefit of transforming the invertible building block into an ODE model is still not convincing to me.
The authors explained that ODE solutions can represent highly nonlinear functions. However, discrete NN also represents highly nonlinear functions (e.g. with Relu activation, they are *piecewise* linear, but they are highly nonlinear)! From my point of view, their difference is that ODE model is more smooth. However, the benefit of using a smoother model is still unclear. For the example that the authors provided, $\sin x$, why being able to represent that kind of function is an advantage for the graph classification problem? Why is this a good model bias? I think the authors' responses are still not convincing enough, so I choose to retain the score.