Method of statistical analysis
Bayesian linear regression is a type of conditional modeling in which the mean of one variable is described by a linear combination of other variables, with the goal of obtaining the posterior probability of the regression coefficients (as well as other parameters describing the distribution of the regressand) and ultimately allowing the out-of-sample prediction of the regressand (often labelled formula_1) "conditional on" observed values of the regressors (usually formula_2). The simplest and most widely used version of this model is the "normal linear model", in which formula_1 given formula_2 is distributed Gaussian. In this model, and under a particular choice of prior probabilities for the parameters—so-called conjugate priors—the posterior can be found analytically. With more arbitrarily chosen priors, the posteriors generally have to be approximated.
Model setup.
Consider a standard linear regression problem, in which for formula_5 we specify the mean of the conditional distribution of formula_6 given a formula_7 predictor vector formula_8:
formula_9
where formula_10 is a formula_7 vector, and the formula_12 are independent and identically normally distributed random variables:
formula_13
This corresponds to the following likelihood function:
formula_14
The ordinary least squares solution is used to estimate the coefficient vector using the Moore–Penrose pseudoinverse:
formula_15
where formula_16 is the formula_17 design matrix, each row of which is a predictor vector formula_18; and formula_19 is the column formula_20-vector formula_21.
This is a frequentist approach, and it assumes that there are enough measurements to say something meaningful about formula_10. In the Bayesian approach, the data are supplemented with additional information in the form of a prior probability distribution. The prior belief about the parameters is combined with the data's likelihood function according to Bayes theorem to yield the posterior belief about the parameters formula_10 and formula_24. The prior can take different functional forms depending on the domain and the information that is available "a priori".
Since the data comprise both formula_19 and formula_16, the focus only on the distribution of formula_19 conditional on formula_16 needs justification. In fact, a "full" Bayesian analysis would require a joint likelihood formula_29 along with a prior formula_30, where formula_31 symbolizes the parameters of the distribution for formula_16. Only under the assumption of (weak) exogeneity can the joint likelihood be factored into formula_33. The latter part is usually ignored under the assumption of disjoint parameter sets. More so, under classic assumptions formula_16 are considered chosen (for example, in a designed experiment) and therefore has a known probability without parameters.
With conjugate priors.
Conjugate prior distribution.
For an arbitrary prior distribution, there may be no analytical solution for the posterior distribution. In this section, we will consider a so-called conjugate prior for which the posterior distribution can be derived analytically.
A prior formula_35 is conjugate to this likelihood function if it has the same functional form with respect to formula_10 and formula_24. Since the log-likelihood is quadratic in formula_10, the log-likelihood is re-written such that the likelihood becomes normal in formula_39. Write
formula_40
The likelihood is now re-written as
formula_41
where
formula_42
where formula_43 is the number of regression coefficients.
This suggests a form for the prior:
formula_44
where formula_45 is an inverse-gamma distribution
formula_46
In the notation introduced in the inverse-gamma distribution article, this is the density of an formula_47 distribution with formula_48 and formula_49 with formula_50 and formula_51 as the prior values of formula_52 and formula_53, respectively. Equivalently, it can also be described as a scaled inverse chi-squared distribution, formula_54
Further the conditional prior density formula_55 is a normal distribution,
formula_56
In the notation of the normal distribution, the conditional prior distribution is formula_57
Posterior distribution.
With the prior now specified, the posterior distribution can be expressed as
formula_58
With some re-arrangement, the posterior can be re-written so that the posterior mean formula_59 of the parameter vector formula_10 can be expressed in terms of the least squares estimator formula_61 and the prior mean formula_62, with the strength of the prior indicated by the prior precision matrix formula_63
formula_64
To justify that formula_59 is indeed the posterior mean, the quadratic terms in the exponential can be re-arranged as a quadratic form in formula_66.
formula_67
Now the posterior can be expressed as a normal distribution times an inverse-gamma distribution:
formula_68
Therefore, the posterior distribution can be parametrized as follows.
formula_69
where the two factors correspond to the densities of formula_70 and formula_71 distributions, with the parameters of these given by
formula_72
formula_73
which illustrates Bayesian inference being a compromise between the information contained in the prior and the information contained in the sample.
Model evidence.
The model evidence formula_74 is the probability of the data given the model formula_75. It is also known as the marginal likelihood, and as the "prior predictive density". Here, the model is defined by the likelihood function formula_76 and the prior distribution on the parameters, i.e. formula_77. The model evidence captures in a single number how well such a model explains the observations. The model evidence of the Bayesian linear regression model presented in this section can be used to compare competing linear models by Bayesian model comparison. These models may differ in the number and values of the predictor variables as well as in their priors on the model parameters. Model complexity is already taken into account by the model evidence, because it marginalizes out the parameters by integrating formula_78 over all possible values of formula_10 and formula_24.
formula_81
This integral can be computed analytically and the solution is given in the following equation.
formula_82
Here formula_83 denotes the gamma function. Because we have chosen a conjugate prior, the marginal likelihood can also be easily computed by evaluating the following equality for arbitrary values of formula_10 and formula_24.
formula_86
Note that this equation is nothing but a re-arrangement of Bayes theorem. Inserting the formulas for the prior, the likelihood, and the posterior and simplifying the resulting expression leads to the analytic expression given above.
Other cases.
In general, it may be impossible or impractical to derive the posterior distribution analytically. However, it is possible to approximate the posterior by an approximate Bayesian inference method such as Monte Carlo sampling, INLA or variational Bayes.
The special case formula_87 is called ridge regression.
A similar analysis can be performed for the general case of the multivariate regression and part of this provides for Bayesian estimation of covariance matrices: see Bayesian multivariate linear regression.
Notes.
 ;templatestyles src="Reflist/styles.css" / ;