Innovation Method
In statistics, the Innovation Method provides an estimator for the parameters of stochastic differential equations given a time series of (potentially noisy) observations of the state variables. In the framework of continuous-discrete state space models, the innovation estimator is obtained by maximizing the log-likelihood of the corresponding discrete-time innovation process with respect to the parameters. The innovation estimator can be classified as a M-estimator, a quasi-maximum likelihood estimator or a prediction error estimator depending on the inferential considerations that want to be emphasized. The innovation method is a system identification technique for developing  mathematical models of  dynamical systems from measured data and for the optimal design of experiments. Background
Stochastic differential equations (SDEs) have become an important mathematical tool for describing the time evolution of several random phenomenon in  natural, social and applied sciences. Statistical inference for SDEs is thus of great importance in applications for model building, model selection, model identification and forecasting. To carry out statistical inference for SDEs, measurements of the state variables of these random phenomena are indispensable. Usually, in practice, only a few state variables are measured by physical devices that introduce random measurement errors (observational errors). Mathematical model for inference
The innovation estimator. for SDEs is defined in the framework of continuous-discrete state space models. These models arise as natural mathematical representation of the temporal evolution of continuous random phenomena and their measurements in a succession of time instants. In the simplest formulation, these continuous-discrete models  are expressed in term
of a SDE of the form

describing the time evolution of  state variables  of the phenomenon for all time instant , and an observation equation

describing the time series of measurements of at least one of the variables  of the random phenomenon on  time instants . In the model (1)-(2),  and  are differentiable functions,  is an  -dimensional standard Wiener process,  is a vector of   parameters,  is a sequence of -dimensional  i.i.d. Gaussian random vectors independent of ,   an  positive definite matrix, and   an  matrix. Statistical problem to solve 
Once the dynamics of a phenomenon is described by a state equation as (1) and the way of measurement the state variables specified by an observation equation as (2), the inference problem to solve is the following: given  partial and noisy observations  of the stochastic process  on the observation times , estimate the unobserved state variable of  and the unknown parameters  in (1) that better fit to the given observations. Discrete-time innovation process 
Let  be the sequence of  observation times  of the states of (1), and  the time series of partial and noisy measurements of  described by the observation equation (2). Further, let  and  be the conditional mean and variance of  with , where  denotes the expected value of random vectors. The random sequence  with 

defines the discrete-time innovation process, where  is proved to be an independent normally distributed random vector with zero mean and variance 

for small enough , with . In practice, this distribution for the discrete-time innovation is valid when, with a suitable selection of both, the number  of observations and the time distance  between consecutive observations, the time series of observations  of the SDE contains the main information about the continuous-time process . That is, when the sampling of the continuous-time process  has low distortion (aliasing) and when there is a suitable signal-noise ratio. Innovation estimator 
The innovation estimator for the parameters of the SDE (1) is the one that maximizes the likelihood function of the discrete-time innovation process  with respect to the parameters. More precisely, given  measurements of the state space model (1)-(2) with   on   the innovation estimator for the parameters  of (1) is defined by

where

being the discrete-time innovation (3) and the innovation variance (4) of the model (1)-(2) at , for all  In the above expression for  the conditional mean  and variance  are  computed by the continuous-discrete filtering algorithm for the evolution of the moments (Section 6.4 in),  for all

Differences with the maximum likelihood estimator 
The maximum likelihood estimator of the parameters  in the model (1)-(2) involves the evaluation of the - usually unknown - transition density function  between the states   and  of the diffusion process  for all the observation times  and . Instead of this, the innovation estimator (5) is obtained by maximizing the likelihood of the discrete-time innovation process  taking into account that  are Gaussian and independent random vectors. Remarkably, whereas the transition density function  changes when the SDE for  does, the transition density function  for the innovation process remains Gaussian independently of the SDEs for . Only in the case that the diffusion  is described by a linear SDE with additive noise, the density function  is Gaussian and equal to  and so the maximum likelihood and the innovation estimator coincide. Otherwise, the innovation estimator is an approximation to the maximum likelihood estimator and, in this sense, the innovation estimator is a Quasi-Maximum Likelihood estimator. In addition, the innovation method is a particular instance of the Prediction Error method according to the definition given in. Therefore, the asymptotic results obtained in for that general class of estimators are valid for the innovation estimators. Intuitively, by following the typical control engineering viewpoint, it is expected that the innovation process - viewed as a measure of the prediction errors of the fitted model - be approximately a white noise process when the models fit the data, which can be used as a practical tool for designing of models and for optimal experimental design. Properties 
The innovation estimator (5) has a number of important attributes:

 Under conventional regularity conditions, the innovation estimator (5) is consistent and asymptoticaly normal distributed. For  selecting models, the maximum log-likelihood  of the innovation estimator (5) can be used to compute the  Akaike or Bayesian information criterion. The   confidence limits  for the innovation estimator  is estimated with
                                                                       

where  is the t-student distribution with  significance level, and  degrees of freedom . Here,  denotes the variance of the innovation estimator , where

is the Fisher Information matrix the innovation estimator  of  and

is the entry  of the matrix  with  and , for . The distribution of the fitting-innovation process  measures the goodness of fit of the model to the data. For smooth enough function , nonlinear observation equations of the form 

can be transformed to the simpler one (2), and the innovation estimator (5) can be applied. Approximate Innovation estimators 

In practice, close form expressions for computing  and  in (5) are only available for a few models (1)-(2). Therefore, approximate filtering algorithms as the following are used in applications. Given  measurements  and the initial filter estimates , , the approximate Linear Minimum Variance (LMV) filter for the model (1)-(2) is iteratively defined at each observation time  by the prediction estimates

  and  

with initial conditions  and , and the filter estimates

  and  

with filter gain

for all , where  is an approximation to the solution  of (1) on the observation times . Given  measurements  of the state space model (1)-(2) with  on , the approximate innovation estimator for the parameters  of (1) is defined by

where 

being 

  and  

approximations to the discrete-time innovation (3) and innovation variance (4), respectively, resulting from the filtering algorithm (7)-(8). For models with complete observations free of noise (i.e., with  and  in (2)), the approximate innovation estimator (9) reduces to the known Quasi-Maximum Likelihood estimators for SDEs. Main conventional-type estimators 

Conventional-type innovation estimators are those (9) derived from conventional-type continuous-discrete or
discrete-discrete approximate filtering algorithms. With approximate continuous-discrete filters there are the innovation estimators based on Local Linearization (LL) filters, on the extended Kalman filter, and on the  second order filters. Approximate innovation estimators based on discrete-discrete filters result from the discretization of the SDE (1) by means of a numerical scheme. Typically, the effectiveness of these innovation estimators is directly related to the  stability of the involved filtering algorithms. A shared drawback of these conventional-type filters is that, once the observations are given, the error between the approximate and the exact innovation process is fixed and completely settled by the time distance between observations. This might set a large  bias of the approximate innovation estimators in some applications, bias that cannot be corrected by increasing the number of observations. However, the conventional-type innovation estimators are useful in many practical situations for which only medium or low  accuracy for the parameter estimation is required. Order-β innovation estimators  

Let us consider the finer time discretization  of the time interval  satisfying the condition . Further, let  be the approximate value of  obtained from a discretization of the equation (1) for all , and 

  for all 

a continuous-time approximation to . A order-  LMV filter. is an approximate LMV filter for which  is an order-  weak approximation to  satisfying (10) and the  weak convergence condition

for all  and any  times continuously differentiable functions  for which  and all its partial derivatives up to order  have polynomial growth, being  a positive constant. This order- LMV filter converges with rate  to the exact LMV filter as  goes to zero, where  is the maximum stepsize of the time
discretization  on which the approximation  to  is defined. A order-  innovation estimator is an approximate innovation estimator (9) for which the approximations to the discrete-time innovation (3) and innovation variance (4), respectively, resulting from an order-  LMV filter. Approximations  of any kind converging to  in a weak sense (as, e.g., those in ) can be used to design an order- LMV filter and, consequently, an order- innovation estimator. These order- innovation estimators are intended for the recurrent practical situation in which a diffusion process should be identified from a reduced number of observations distant in time or when high accuracy for the estimated parameters is required. Properties  

An order- innovation estimator  has a number of important properties:

 For each given data  of  observations,  converges to the exact innovation estimator  as the maximum stepsize  of the time discretization  goes to zero. For finite samples of  observations, the expected value of  converges to the expected value of the exact innovation estimator  as  goes to zero. For an increasing number of observations,  is asymptotically normal distributed and its bias decreases when  goes to zero. Likewise to the convergence of the order- LMV filter to the exact LMV filter, for the convergence and asymptotic properties of  there are no constraints on the time distance  between two consecutive observations  and , nor on the time discretization 

 Approximations for the Akaike or Bayesian information criterion and confidence limits are directly obtained by replacing the exact estimator  by its approximation . These approximations converge to the corresponding exact one when the maximum stepsize  of the time discretization  goes to zero. The distribution of the approximate fitting-innovation process  measures the goodness of fit of the model to the data, which is also used as a practical tool for designing of models and for optimal experimental design. For smooth enough function , nonlinear observation equations of the form (6) can be transformed to the simpler one (2), and the order- innovation estimator can be applied. Figure 1 presents the histograms of the differences  and  between the exact innovation estimator  with the conventional  and order-  innovation estimators for the parameters  and  of the equation

obtained from 100 time series  of  noisy observations

of  on the observation times , , with  and . The classical and the order- Local Linearization filters of the innovation
estimators  and   are defined as in, respectively, on the uniform time discretizations  and , with . The number of stochastic simulations of the order- Local Linearization filter is estimated via an adaptive sampling algorithm with moderate tolerance. The Figure 1 illustrates the convergence of the order- innovation estimator  to the exact innovation estimators  as  decreases, which substantially improves the estimation provided by the conventional innovation estimator . Deterministic approximations  

The order- innovation estimators overcome the drawback of the conventional-type innovation estimators concerning the impossibility of reducing bias. However, the viable bias reduction of an order- innovation estimators might eventually require that the associated order- LMV filter performs a large number of stochastic simulations. In situations where only low or medium precision approximate estimators are needed, an alternative deterministic filter algorithm - called deterministic order- LMV filter  - can be obtained by tracking the first two conditional moments  and  of the order- weak approximation  at all the time instants  in between two consecutive observation times  and . That is, the value of the predictions  and  in the filtering algorithm are computed from the recursive formulas

  and    with 

and with . The approximate innovation estimators  defined with these deterministic order- LMV filters not longer converge to the exact innovation estimator, but allow a significant bias reduction in the estimated parameters for a given finite sample with a lower computational cost. Figure 2 presents the histograms and the confidence limits of the approximate innovation estimators  and  for the parameters  and  of the Van der Pol oscillator with random frequency

obtained from 100 time series  of  partial and noisy observations

of  on the observation times , , with  and . The deterministic order- Local Linearization filter of the innovation estimators  and  is defined, for each estimator, on uniform time discretizations , with  and on an adaptive time-stepping discretization  with moderate relative and absolute tolerances, respectively. Observe the bias reduction of the estimated parameter as  decreases. Software 

A Matlab implementation of various approximate innovation estimators is provided by the SdeEstimation toolbox. This toolbox has Local Linearization filters, including deterministic and stochastic options with fixed step sizes and sample numbers. It also offers adaptive time stepping and sampling algorithms, along with local and global optimization algorithms for innovation estimation. For models with complete observations free of noise, various approximations to the Quasi-Maximum Likelihood estimator are implemented in R.

References

Estimation methods