Regression models accounting for possible errors in independent variables
In statistics, an errors-in-variables model or a measurement error model is a regression model that accounts for measurement errors in the independent variables. In contrast, standard regression models assume that those regressors have been measured exactly, or observed without error; as such, those models account only for errors in the dependent variables, or responses.
In the case when some regressors have been measured with errors, estimation based on the standard assumption leads to inconsistent estimates, meaning that the parameter estimates do not tend to the true values even in very large samples. For simple linear regression the effect is an underestimate of the coefficient, known as the "attenuation bias". In non-linear models the direction of the bias is likely to be more complicated.
Motivating example.
Consider a simple linear regression model of the form
formula_1
where formula_2 denotes the "true" but unobserved regressor. Instead we observe this value with an error:
formula_3
where the measurement error formula_4 is assumed to be independent of the true value formula_2.
 ;br ;A practical application is the standard school science experiment for Hooke's Law, in which one estimates the relationship between the weight added to a spring and the amount by which the spring stretches. ;br ;
If the formula_6′s are simply regressed on the formula_7′s (see simple linear regression), then the estimator for the slope coefficient is
 formula_8
which converges as the sample size formula_9 increases without bound:
 formula_10
This is in contrast to the "true" effect of formula_11, estimated using the formula_2,:
 formula_13
Variances are non-negative, so that in the limit the estimated formula_14 is smaller than formula_15, an effect which statisticians call "attenuation" or regression dilution. Thus the ‘naïve’ least squares estimator formula_14 is an inconsistent estimator for formula_11. However, formula_14 is a consistent estimator of the parameter required for a best linear predictor of formula_19 given the observed formula_20: in some applications this may be what is required, rather than an estimate of the ‘true’ regression coefficient formula_11, although that would assume that the variance of the errors in the estimation and prediction is identical. This follows directly from the result quoted immediately above, and the fact that the regression coefficient relating the formula_6′s to the actually observed formula_7′s, in a simple linear regression, is given by
 formula_24
It is this coefficient, rather than formula_11, that would be required for constructing a predictor of formula_19 based on an observed formula_27 which is subject to noise.
It can be argued that almost all existing data sets contain errors of different nature and magnitude, so that attenuation bias is extremely frequent (although in multivariate regression the direction of bias is ambiguous). Jerry Hausman sees this as an "iron law of econometrics": "The magnitude of the estimate is usually smaller than expected."
Specification.
Usually measurement error models are described using the latent variables approach. If formula_19 is the response variable and formula_27 are observed values of the regressors, then it is assumed there exist some latent variables formula_30 and formula_31 which follow the model's “true” functional relationship formula_32, and such that the observed quantities are their noisy observations:
 formula_33
where formula_34 is the model's parameter and formula_35 are those regressors which are assumed to be error-free (for example when linear regression contains an intercept, the regressor which corresponds to the constant certainly has no "measurement errors"). Depending on the specification these error-free regressors may or may not be treated separately; in the latter case it is simply assumed that corresponding entries in the variance matrix of formula_36's are zero.
The variables formula_19, formula_27, formula_35 are all "observed", meaning that the statistician possesses a data set of formula_40 statistical units formula_41 which follow the data generating process described above; the latent variables formula_42, formula_43, formula_44, and formula_36 are not observed however.
This specification does not encompass all the existing errors-in-variables models. For example in some of them function formula_32 may be non-parametric or semi-parametric. Other approaches model the relationship between formula_43 and formula_42 as distributional instead of functional, that is they assume that formula_43 conditionally on formula_42 follows a certain (usually parametric) distribution.
Linear model.
Linear errors-in-variables models were studied first, probably because linear models were so widely used and they are easier than non-linear ones. Unlike standard least squares regression (OLS), extending errors in variables regression (EiV) from the simple to the multivariable case is not straightforward, unless one treats all variables in the same way i.e. assume equal reliability.
Simple linear model.
The simple linear errors-in-variables model was already presented in the "motivation" section:
 formula_67
where all variables are scalar. Here "α" and "β" are the parameters of interest, whereas "σε" and "ση"—standard deviations of the error terms—are the nuisance parameters. The "true" regressor "x*" is treated as a random variable ("structural" model), independent of the measurement error "η" ("classic" assumption).
This model is identifiable in two cases: (1) either the latent regressor "x*" is "not" normally distributed, (2) or "x*" has normal distribution, but neither "εt" nor "ηt" are divisible by a normal distribution. That is, the parameters "α", "β" can be consistently estimated from the data set formula_68 without any additional information, provided the latent regressor is not Gaussian.
Before this identifiability result was established, statisticians attempted to apply the maximum likelihood technique by assuming that all variables are normal, and then concluded that the model is not identified. The suggested remedy was to "assume" that some of the parameters of the model are known or can be estimated from the outside source. Such estimation methods include
Estimation methods that do not assume knowledge of some of the parameters of the model, include
Multivariable linear model.
The multivariable model looks exactly like the simple linear model, only this time "β", "η""t", "x""t" and "x*""t" are "k×"1 vectors.
 formula_69
In the case when ("ε""t","η""t") is jointly normal, the parameter "β" is not identified if and only if there is a non-singular "k×k" block matrix ["a A"], where "a" is a "k×"1 vector such that "a′x*" is distributed normally and independently of "A′x*". In the case when "ε""t", "η""t1"..., "η""tk" are mutually independent, the parameter "β" is not identified if and only if in addition to the conditions above some of the errors can be written as the sum of two independent variables one of which is normal.
Some of the estimation methods for multivariable linear models are
Non-linear models.
A generic non-linear measurement error model takes form
 formula_70
Here function "g" can be either parametric or non-parametric. When function "g" is parametric it will be written as "g"("x"*, "β").
For a general vector-valued regressor "x*" the conditions for model identifiability are not known. However in the case of scalar "x*" the model is identified unless the function "g" is of the "log-exponential" form 
 formula_71
and the latent regressor "x*" has density
 formula_72
where constants "A","B","C","D","E","F" may depend on "a","b","c","d".
Despite this optimistic result, as of now no methods exist for estimating non-linear errors-in-variables models without any extraneous information. However there are several techniques which make use of some additional data: either the instrumental variables, or repeated observations.
Repeated observations.
In this approach two (or maybe more) repeated observations of the regressor "x*" are available. Both observations contain their own measurement errors, however those errors are required to be independent:
 formula_73
where "x*" ⊥ "η"1 ⊥ "η"2. Variables "η"1, "η"2 need not be identically distributed (although if they are efficiency of the estimator can be slightly improved). With only these two observations it is possible to consistently estimate the density function of "x*" using Kotlarski's deconvolution technique.
\big( x_{j} - x^*_{j} \big),
 ;/math ;
where with slight abuse of notation "xj" denotes the "j"-th component of a vector. ;br ;
All densities in this formula can be estimated using inversion of the empirical characteristic functions. In particular,
 formula_74
In order to invert these characteristic function one has to apply the inverse Fourier transform, with a trimming parameter "C" needed to ensure the numerical stability. For example:
 formula_75
 formula_76
where "wt" represents variables measured without errors. The regressor "x*" here is scalar (the method can be extended to the case of vector "x*" as well). ;br ;
If not for the measurement errors, this would have been a standard linear model with the estimator
 formula_77
where
formula_78
It turns out that all the expected values in this formula are estimable using the same deconvolution trick. In particular, for a generic observable "wt" (which could be 1, "w"1"t", …, "w""ℓ  t", or "yt") and some function "h" (which could represent any "gj" or "gigj") we have
 formula_79
where "φh" is the Fourier transform of "h"("x*"), but using the same convention as for the characteristic functions,
 formula_80,
and
 formula_81
The resulting estimator formula_82 is consistent and asymptotically normal.
 formula_83
for a suitable choice of the kernel "K" and the bandwidth "h". Both expectations here can be estimated using the same technique as in the previous method.
References.
 ;templatestyles src="Reflist/styles.css" / ;