Neural Ordinary Differential Equations
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a blackbox differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.
# Response to author feedback  My thanks to the authors for their responses to my comments and questions in their feedback and commitment to make several clarifications in response to the suggestions made. After reading their feedback and the other reviewer's comments I maintain my original assessment and feel this is an excellent submission that I'd strongly argue for including in the conference.  ---  # Summary and relation to previous work  This submission makes several related contributions:  * it provides a clear and readable introduction to the adjoint sensitivity methods for reverse-mode automatic differentiation (AD) through ODE solvers,  * motivated by previous links drawn between Res-Nets and ODE models (Lu, Zhong, Li & Dong, 2017), the direct use of ODE solvers as differentiable (and so learnable) components in machine learning models is proposed, with such 'ODE-Nets' offering advantages of improved memory efficiency during gradient calculation and adpative effective network depth,  * continuous normalising flows (CNFs) are proposed as an analogue to previous work on flow models e.g. normalising flows (Rezende & Mohamed, 2015) and NICE (Dinh, Krueger and Bengio, 2015), with CNFs claimed to provided the advantage of allowing more expressive flows while maintaining computational tractability and efficient reversibility of the flow (compared to normalising flows),   * a generative latent variable ODE-based time series model (L-ODE) trained with a variational objective is proposed as an alternative to standard recurrent neural network approaches, with more natural handling of irregularly time-sampled sequences.  # Strengths and weaknesses  Although the adjoint sensitivity method is an existing method, exposing this method to machine learning and computational statistics communities where, as far as I am aware it is not widely known about, is a worthwile contribution of this submission its own right. Given the ever increasing importance of AD in both communities, adding to the range of scientific computing primitives for which frameworks such as autograd can efficiently compute derivatives through will hopefully spur more widespread use of gradient based learning and inference methods with ODE models and hopefully spur other frameworks with AD capability in the community such as Stan, TensorFlow and Pytorch to implement adjoint sensitivity methods.  The specific suggested applications of the 'ODE solver modelling primitive' in ODE-Nets, CNFs and L-ODEs are all interesting demonstrations of some of the computational and modelling advantages that come from using a continuous-time ODE mode; formulation, with in particular the memory savings possible by avoiding the need to compute all intermediate states by recomputing trajectories backwards through time being a possible major gain given that device memory is often currently a bottleneck. While 'reversing' the integration to recompute the reverse trajectory is an appealing idea, it would have helped to have more discussion of when this would be expected to breakdown - for example it seems likely that highly chaotic dynamical systems would tend to be problematic as even small errors in the initial backwards steps could soon lead to very large divergences in the reversed trajectories compared to the forward ones. It seems like a useful sanity check in an implementation would be to compare the final state of the reversed trajectory to the initial state of the forward trajectory to check how closely they agree.  The submission is generally very well written and presented with a clear expository style, with useful illustrative examples given in the experiments to support the claims made and well thought out figures which help to give visual intuitions about the methods and results. There is a lot of interesting commentary and ideas in the submission with there seeming to be a lot of potential in even side notes like the concurrent mixutre of dynamics idea. While this makes for an interesting and thought-provoking read, the content-heavy nature of the paper and slightly rambling exploration of many ideas are perhaps not ideally suited to such a short conference paper format, with the space constraints meaning sacrifices have been made in terms of the depth of discussion of each idea, somewhat terse description of the methods and results in some of the experiments and in some cases quite cramped figure layouts. It might be better to cull some of the content or move it to an appendix to make the main text more focussed and to allow more detailed discussion of the remaining areas.  A more significant weakness perhaps is a lack of empirical demonstrations on larger benchmark problems for either the ODE-Nets or CNFs to see how / if the proposed advantages over Res-Nets and NFs respectively carry over to (slightly) more realistic settings, for example using the CNF in a VAE image model on MNIST / CIFAR-10 as in the final experiments in original NF paper. Although I don't think such experiments are vital given the current numerical experiments do provide some validation of the claims already and more pragmatically given that the submission already is quite content heavy already so space is a constaint, some form of larger scale experiments would make a nice addition to an already strong contribution.  # Questions   * Is the proposal to backward integrate the original ODE to allow access to the (time-reversed) trajectory when solving the adjoint ODE rather than storing the forward trajectory novel or is this the standard approach in implementations of the method?  * Does the ResNet in the experiments in section 7.1 share parameters between the residual blocks? If not a potentially further interesting baseline for the would be to compare to a deeper ResNet with parameter sharing as this would seem to be equivalent to an ODE net with a fixed time-step Euler integrator.  * What is the definition used for the numerical error on the vertical axis in Figure 4a and how is the 'truth' evaluated?  * How much increase in computation time (if any) is there in computing the gradient of a scalar loss based on the output of `odeint` compared to evaluating the scalar loss itself using your Python `autograd` implementation in Appendix C? It seems possible that the multiple sequential calls to the `odeint` function between pairs of successive time points when calculating the gradient may introduce a lot of overhead compared to a single call to `odeint` when calculating the loss itself even if the overall number of inner integration steps is similar? Though even if this is the case this could presumably be overcome with a more efficient implementation it would be interesting to get a ballpark for how quick the gradient calculation is currently.  # Minor comments / suggestions / typos:  Theorem 1 and proof in appendix B: while the derivation of this result in the appendix is nice to have as an additional intuition for how the expression arises, it seems this might be more succinctly seen as direct implication of the Fokker-Planck equation for a zero noise process or equivalently from Liouville's equation (see e.g. Stochastic Methods 4th ed., Gardiner, pg. 54). Similarly expressing in terms of the time derivative of the density rather than log density would perhaps make the analogy to the standard change of variables formula more explicit.  Equation following line 170: missing right parenthesis in integrand of last integral.  Algorithm 1: Looks like a notation change might have lead to some inconsistencies - terms subscripted with non-defined $N$: $s_N$, $t_N$, look like they would be more consistent if instead subscripted with 1. Also on first line vector 0 and scalar 0 are swapped in $s_N$ expression  References: some incorrect capitalisation in titles and inconsistent used of initials rather than full first names in some references.  Appendices  L372: 'differentiate by parts' -> 'integrate by parts'  L385: Unclear what is meant by 'the function $j(x,\theta,t)$ is unknown - for the standard case of a loss based on a sum of loss terms each depending on state at a finite set of time points, can we not express $j$ as something like  $$ j(x,\theta,t) = \sum_{k=1}^K \delta(t_k - t) \ell_k(x, \theta) $$  which we can take partial derivatives of wrt $x$ and then directly subsitute into equation (19)?   L394: 'adjoing' -> 'adjoint'