Algorithmic Linearly Constrained Gaussian Processes
We algorithmically construct multi-output Gaussian process priors which satisfy linear differential equations. Our approach attempts to parametrize all solutions of the equations using Gröbner bases. If successful, a push forward Gaussian process along the paramerization is the desired prior. We consider several examples from physics, geomathematics and control, among them the full inhomogeneous system of Maxwell's equations. By bringing together stochastic learning and computer algebra in a novel way, we combine noisy observations with precise algebraic computations.
The paper addresses the problem of constructing GP priors satisfying constraints that can be expressed as algebraic problems. They demonstrate how this is facilitated by the application of Groebner bases. They discuss some of the mathematical prorperties and show example applications.  I will first address the biggest weakness of the paper: its clarity. From the presentation, it seems evident that the author is an expert in the field of computer algebra/algebraic geometry. It is my assumption that most members of the NIPS community will not have a strong background on this subject, me including. As a consequence, I found it very hard to follow Sect. 3. My impression was that the closer the manuscript comes to the core of algebraic geometry results, the less background was provided. In particular, I would have loved to see at least a proof idea or some more details/background on Thm. 3.1 and Cor. 3.2. Or maybe, the author could include one less example in the main text but show the entire derivation how to get from one concrete instance of A to right kernel B by manual computation? Also, for me the description in Sect. 2.4 was insufficient. As a constructive instruction, maybe drop one of the examples (R(del_t) / R[sigma_x]), but give some more background on the other?  This problem of insufficient clarity cannot be explained by different backgrounds alone. In Sect. 3.2, the sentence "They are implemented in various computer algebra systems, 174 e.g., Singular [8] and Macaulay2 [16] are two well-known open source systems." appears twice (and also needs grammar checking). If the author could find a minimal non-trivial example (to me, this would be an example not including the previously considered linear differential operator examples) for which the author can show the entire computation in Sect. 3.2 or maybe show pseudo-code for some algorithms involving the Groebner basis, this would probably go a long way in the community.  That being said, the paper's strengths are (to the best of this reviewer's knowledge) its originality and potential significance. The insight that Groebner bases can be used as a rich language to encode algebraic constraints and highlighting the connection to this vast background theory opens an entirely new approach in modelling capacities for Gaussian processes. I can easily imagine this work being the foundation for many physical/empirical-hybrid models in many engineering applications. I fully agree and applaud the rationale in lines 43-54!  Crucially, the significance of this work will depend on whether this view will be adopted fast enough by the rest of the community which in turn depends on the clarity of the presentation.  In conclusion: if I understood the paper correctly, I think the theory presented therein is highly original and significant, but in my opinion, the clarity should be improved significantly before acceptance, if this work should reach its full potential. However, if other reviewers have a different opinion on the level of necessary background material, I would even consider this work for oral presentation.  Minor suggestions for improvements: - In line 75, the author writes that the "mean function is used as regression model" and this is how the author uses GPs throughout. However, in practice the (posterior) covariance is also considered as "measure of uncertainty". It would be insightful, if the author could find a way to visualize this for one or two of the examples the author considers, e.g., by drawing from the posterior process. - I am not familiar with the literature: all the considerations in this paper should also be applicable to kernel (ridge) regression, no? Maybe this could also be presented in the 'language of kernel interpolation/smoothing' as well? - I am uncertain about the author's reasoning on line 103. Does the author want to express that the mean is a sample from the GP? But the mean is not a sample from the GP with probability 1. Generally, there seems to be some inconsistency with the (algebraic) GP object and samples from said object. - The comment on line 158 "This did not lead to practical problems, yet." is very ominous. Would we even expect any problem? If not, I would argue you can drop it entirely. - I am not sure whether I understood Fig. 2 correctly. Am I correct that u(t) is either given by data or as one draw from the GP and then, x(t) is the corresponding resulting state function for this specified u? I'm assuming that Fig. 3 is done the other way around, right? --- Post-rebuttal update: Thank you for your rebuttal. I think that adding computer-algebra code sounds like a good idea.  Maybe presenting the work more in the context of kernel ridge regression would eliminate the discussion about interpreting the uncertainty. Alternatively, if the author opts to present it as GP, maybe a video could be used to represent the uncertainty by sampling a random walk through the distribution.  Finally, it might help to not use differential equations as expository material. I assume the author's rationale for using this was that reader might already a bit familiar with it and thus help its understanding. I agree, but for me it made it harder to understand the generality with respect to Groebner bases. My first intuition was that "this has been done". Maybe make they Weyl algebra and Figure 4 the basic piece? But I expect this suggestion to have high variance.