Implicit Probabilistic Integrators for ODEs
We introduce a family of implicit probabilistic integrators for initial value problems (IVPs), taking as a starting point the multistep Adams-Moulton method. The implicit construction allows for dynamic feedback from the forthcoming timestep, in contrast to previous probabilistic integrators, all of which are based on explicit methods. We begin with a concise survey of the rapidly-expanding field of probabilistic ODE solvers. We then introduce our method, which builds on and adapts the work of Conrad et al. (2016) and Teymur et al. (2016), and provide a rigorous proof of its well-definedness and convergence. We discuss the problem of the calibration of such integrators and suggest one approach. We give an illustrative example highlighting the effect of the use of probabilistic integrators-including our new method-in the setting of parameter inference within an inverse problem.
## Summary  The paper proposes a novel algorithm for solving initial value problems in ODEs. The first part of the paper aims to review current literature on probabilistic ODE solvers arising in the field of probabilistic numerics (PN). Then, based on this review, the authors propose a novel probabilistic implicit solver for IVPs which retains some of the non-linearities of the vector field, but locally works with Gaussian approximations to the solution and/or the velocities for computational speedups. The paper contains a proof of convergence in mean square for diminishing step sizes. As a proof-of-concept, the authors demonstrate their solver as well as its application to system identification on a 2D-toy-problem.  ## Novelty & originality & Significance The paper is an interesting read since it is a combination of a review paper and a new proposed method. Apart from the latter, personally I think it has a high values since it summarizes the state-of-the-art of the field well, also for non-ODE experts. Additionally the proposed method (*implicit* rather than explicit PN solver) seems novel to me. It is an interesting approach to handling the cost-precision tradeoff for ODEs which more generally is inherent to any numerical solver. I am not super convinced about the practical relevance of the method in its current state and I’d encourage the authors to comment on the possible applications, especially in contrast to running a faster solver with a smaller step h for the same computational budget.  ## Clarity The paper is clearly written and well developed. It is however very dense. I’d actually like to see a longer journal version of this, too.  ## Major points - Eq. 3 and paragraph below. I do not see the point why Z, Z-tilde and x are named differently since, as I understand, all three represent the same random variable (the unknown solution).    - Eq. 4: the authors mention that their method could in principle use different transformations g but only the Gaussian is used. Did the authors experiment with other transformations? If yes, which? If not, do the authors think that for all practical purposes one might always be restricted to use Gaussian forms of g?   - a general point on Gaussians: the authors seem to point out repeatedly that some distributions are non-Gaussian. This is fair enough to distinguish their method from methods that do use Gaussians in all distributions. But for a reader it is sometimes harder to understand the novel method by having negative statements only (what it is not, rather than what it is). I’d encourage the authors to also point out more clearly which of the latent variables (F for instance? Or r?) are Gaussian locally, especially since Gaussians seem to appear later during the approximations and estimation of H. It would make for an easier read.  - l. 242 “… nowhere assuming that the distributions are Gaussian.” Related to the above point, this comment is a bit misleading and prone to misunderstandings since it is not clear what “the distributions” means. I’d encourage the authors to specify which distribution they were referring to, especially since Gaussian distributions do occur in various places.  - l. 270. “Then non-linearities from f are retained (to second order)” I am unsure, but isn’t the approximation linear in f but non-linear in z? At least l. 265 seems to indicate this.  - l. 271 “jointly Gaussian”. What is meant by “jointly” here? Is it jointly across iterations i? Or jointly given one specific i? The former is true, the latter is not I guess. Could the authors please specify.   - Eq. 10: H seems to be fitted in a kind of empirical Bayes. Could the authors comment on this, please. Especially in connection to the idea of “numerical data F”.  - A general point on applicability: As a potential user of the algorithms, what would be the application where the proposed algorithms is better than any existing ODE solver. And also better in which sense? I am asking because the authors motivate their algorithms also by the accuracy-cost tradeoff, and it left me wondering what I would actually use it for. I could, for example also just run a classic ODE solver with a finer step size h and, in the same time, have such a good approximation to my solution that I do not care very much about potential non-symmetric error bars.  - Experiments: please explain which of the approximations are used to generate the results. For instance I think the authors did not mention if the approximations of section 2.4 are used. Also how does the sampling exactly work. Could the authors please add a pseudo-code to the supplements.  - Figure 1 & experiments: Once given the empirical distribution (red dots), is it possible to generate an estimator for the solution? If yes, how? What would be the most favorable choice and why?  ## Minor points - l. 101 and l. 135 “This approach…” Both times a new paragraph is started with a reference to something mentioned before. Possibly the paragraphs can be divided somewhere else such that paragraphs contain one thought process completely. Then tracing back and referencing would become a lot easier.  - Figure 2. Could the authors please add an indication of the true value to the x-axis? Like this one has to trace-back to find the true value of theta in the text.  ## Post rebuttal Thank you for clarifying and addressing my concerns, especially also all the detailed comments [3].  [1] purpose of work: thank you for clarifying. That makes sense to me, although my question was more targeted towards specific examples or applications where, as you say, the richer UQ is preferred over tighter worst-case errors. I understand the high-level argument but would be interested in knowing an application where this trade-off is indeed in favor of the PN-ODE solver. If the authors are aware of one it might be worth mentioning it in the final version. In my mind, a less well-understood but small error is usually preferred over a well-understood, but larger error.  [2] notation Schober et al.: I see your point here. I’d still argue that it is worth mentioning that Z and \tilde{Z} represent the same RV and the distinction in notation only emphasizes the difference in their use. I am kind of picking on this since distinguishing the two is a bit counter-intuitive from a PN point of view and possibly misleading.  [3] all other questions: thanks for clarifying and addressing all the points. It is clearer now.