Scalable Variational Inference for Dynamical Systems
Gradient matching is a promising tool for learning parameters and state dynamics of ordinary differential equations. It is a grid free inference approach, which, for fully observable systems is at times competitive with numerical integration. However, for many real-world applications, only sparse observations are available or even unobserved variables are included in the model description. In these cases most gradient matching methods are difficult to apply or simply do not provide satisfactory results. That is why, despite the high computational cost, numerical integration is still the gold standard in many applications. Using an existing gradient matching approach, we propose a scalable variational inference framework which can infer states and parameters simultaneously, offers computational speedups, improved accuracy and works well even under model misspecifications in a partially observable system.
This paper proposes a mean-field method for joint state and parameter inference of ODE systems. This is a problem that continues to receive attention because of its importance in many fields (since ODEs are used so widely) and the limitations of existing gradient-based methods. The solution described in the paper is said to outperform other approaches, be more applicable, and scale better. 
The paper lays out its contribution clearly in the introduction. It does a good job of discussing the merits and drawbacks of previous approaches, and explaining how it fits in this context. The proposed methodology appears sound (although it might be useful to explain some of the derivations in a little more detail) and the results indicate that it performs much better than the state of the art. The main advantage of the new method is its execution time, which is shown to be orders of magnitude faster than methods with comparable (or worse) accuracy. In particular, the scalability analysis appears very promising for the applicability of this method to larger models.
Overall, I found this to be a strong paper. My comments mainly concern the presentation, and most are minor.
- The biggest comment has to do with the derivations of the evidence lower bounds and Equation (16). I believe it would be useful to have some more details there, such as explicitly pointing out which properties they rely on. It is also not fully clear to me how the E-M steps of Algorithm 1 make use of the previous derivations (although that may be because I have limited experience in this field). There appears to be some space available for expansion but, if necessary, I suppose that sections 2 and 3 could be combined.
- Besides that, there is a statement that appears contradictory. Lines 82-86 describe how parameters and state are alternately sampled in Calderhead et al. and Dondelinger et al. However, lines 29-30 make it seem like these approaches cannot infer both state and parameters. I would suggest either to rephrase (perhaps remove the "simultaneously"?) or explain why this is a problem.
- As far as the experiments are concerned, I think it would be interesting to run the Lotka-Volterra example of Section 5.1 for longer, to confirm that the inferred state displays periodicity (perhaps comparing with the other methods as well). However, I believe the results do a good job of supporting the authors' claims.
Minor comments:
--------------
l2: grid free -> grid-free (also later in the text)
l52: the phrasing makes it sound like the error has zero mean and zero variance; perhaps rephrase to clarify that the variance can be generally elliptical / different for each state?
Equations 3 - 8: There is a small change in the notation for normal distributions between Eqs 3-4 and 5-8. For the sake of clarity, I think it is worth making it consistent, e.g. by changing the rhs of (3) to N(x_k | 0, C) instead of N(0,C) (or the other way around).
l71: What is the difference between C' and 'C ?
Equation 9: no upper bound on the sum
l99: contains -> contain
l99: "arbitrary large" makes me think that a sum term can include repetitions of an x_j (to effectively have power terms, as you would have in mass-action kinetics with stoichiometry coefficients > 1), but it is not fully clear. Is this allowed? (in other words, is M_{ki} a subset of {1,...,K}, or a multiset of its elements?)
l139: what does "number of states" refer to here? the number of terms in the sum in Eq 9?
Figure 1: in the centre-right plot, it seems like the estimate of \theta_4 is missing for the spline method
l155: Predator is x_2, prey is x_1
l163: move the [ after the e.g. : "used in e.g. [Calderhead et al. ..."
l168: usining -> using
l171: I would add commas around "at roughly 100 times the runtime" to make the sentence read easier
l175: partial -> partially
l187: (3) -> (Figure 3)
l195-196: Figure 2 does not show the noise changing; I think having two references would be more helpful: "[...] if state S is unobserved (Figure 2) and [...] than our approach (Figure 3)."
l198: perhaps "error" is better than "bias" here? Unless it is known how exactly the misspecification biases the estimation?
l202: Lorentz -> Lorenz (and later)
l207: stochstic -> stochastic
l208: number states -> number of states
l209: Due the dimensionality -> Due to the dimensionality,
l212: equally space -> equally spaced
l214: correct inferred -> correctly inferred
Figure 4: In the last two lines of the caption: wheras -> whereas; blue -> black (at least that's how it looks to me); is corresponding to -> corresponds to
l219: goldstandard -> gold standard
l253: modemodel -> modelling