Learning Stationary Time Series using Gaussian Processes with Nonparametric Kernels
We introduce the Gaussian Process Convolution Model (GPCM), a two-stage nonparametric generative procedure to model stationary signals as the convolution between a continuous-time white-noise process and a continuous-time linear filter drawn from Gaussian process. The GPCM is a continuous-time nonparametricwindow moving average process and, conditionally, is itself a Gaussian process with a nonparametric kernel defined in a probabilistic fashion. The generative model can be equivalently considered in the frequency domain, where the power spectral density of the signal is specified using a Gaussian process. One of the main contributions of the paper is to develop a novel variational freeenergy approach based on inter-domain inducing variables that efficiently learns the continuous-time linear filter and infers the driving white-noise process. In turn, this scheme provides closed-form probabilistic estimates of the covariance kernel and the noise-free signal both in denoising and prediction scenarios. Additionally, the variational inference procedure provides closed-form expressions for the approximate posterior of the spectral density given the observed data, leading to new Bayesian nonparametric approaches to spectrum estimation. The proposed GPCM is validated using synthetic and real-world signals.
The paper is concerned with the important problem of analyzing impulse responses of LTI systems with GPs. I feel that the paper is nice, but it does not really project the methods to existing approaches which would be an important thing to do. This is a classical problem anyhow. Some comments:- In Section 2 the authors talk about "convergence" without any reference to the mode of convergence or references to literature. Certainly the convergence properties of the integral (1) are known, e.g., in suitable Schwartz spaces because Wiener alone wrote tens of books on harmonic analysis.- Based on the text I would guess that the authors wish to talk about mean square convergence, which is related to Ito's work (stochastic calculus) and where the integral (1) needs to be defined as an Ito integral f(t) = int h(t - tau) dW(tau) which is how it is classically done (see, e.g., the refs. below). - The problem considered here is called "system identification" (cf. google), and it certainly is not a new problem. This problem is classically treated in the framework of "Kalman filtering", which is "GP regression for time series" and works in the Ito calculus / state-space formalism. Nice classic textbooks on the core subject are, e.g., the following:"A. H. Jazwinski (1970). Stochastic processes and filtering theory." "P. Maybeck (1979). Stochastic models, estimation, and control. Volumes 1&2."- The basic classical method is indeed to use a parametric model for the impulse response h(t) instead of a non-parametric model. Typically this impulse response is constructed in a linear state-space form which is then parametrized suitably. State-space methods such as Kalman filtering based approaches can then be used for parameter estimation directly although the resulting model can be treated with other types of methods as well. To clarify: the Kalman filter can be used to evaluate the marginal likelihood using the prediction error decomposition and conventional parameter inference can be implemented on top of it.- It would be beneficial to compare at least to a very simple parametric model to the impulse response---or at least discuss them. You could, for example, use an all-pole type of transfer function modelH(s) = b0 / (a0 + a1 s + a2 s^2)where the parameters a0,a1,a2,b0 are the free parameters. The inverse Laplace transform of this then implicitly defines the impulse response although it is easier to just convert this into state space form and use a Kalman filter type of method to estimate b0,a0,a1,a2. A more elaborate model would be a higher order rational function model.- One point is that DSP books such as Oppenheim et al. does not treat the above kind of models, but you need to look into (oldish) control engineering and system identification literature to find them---although some newer literature might exist. Just to point out that using the classical state-space/Ito approach above, the non-uniform sampling is *not* a problem at all even though from Oppenheim et al it might seem to be.- However, system identification problems are indeed classically parametric and it is a difference to here and the novelty of the article. Furthermore, the above approach for system identification has number of difficulties, for example, it is quite hard to enforce the transfer function to correspond to a stable system. Thus new methods are welcome. In any case, it is not an excuse to ignore the existing literature, you should at least discuss them.- The approach itself is very simple which is nice. It also points out many connections to inducing point methods which is nice.- The weakness in the experiment section is also the lack of comparison to classical approaches.