Function related to statistics and probability theory
A likelihood function (often simply called the likelihood) measures how well a statistical model explains observed data by calculating the probability of seeing that data under different parameter values of the model. It is constructed from the joint probability distribution of the random variable that (presumably) generated the observations. When evaluated on the actual data points, it becomes a function solely of the model parameters.
In maximum likelihood estimation, the argument that maximizes the likelihood function serves as a point estimate for the unknown parameter, while the Fisher information (often approximated by the likelihood's Hessian matrix at the maximum) gives an indication of the estimate's precision.
In contrast, in Bayesian statistics, the estimate of interest is the "converse" of the likelihood, the so-called posterior probability of the parameter given the observed data, which is calculated via Bayes' rule.
Definition.
The likelihood function, parameterized by a (possibly multivariate) parameter formula_1, is usually defined differently for discrete and continuous probability distributions (a more general definition is discussed below). Given a probability density or mass function
formula_2
where formula_3 is a realization of the random variable formula_4, the likelihood function is
formula_5
often written
formula_6
In other words, when formula_7 is viewed as a function of formula_3 with formula_1 fixed, it is a probability density function, and when viewed as a function of formula_1 with formula_3 fixed, it is a likelihood function. In the frequentist paradigm, the notation formula_7 is often avoided and instead formula_13 or formula_14 are used to indicate that formula_1 is regarded as a fixed unknown quantity rather than as a random variable being conditioned on. 
The likelihood function does "not" specify the probability that formula_1 is the truth, given the observed sample formula_17. Such an interpretation is a common error, with potentially disastrous consequences (see prosecutor's fallacy).
Discrete probability distribution.
Let formula_4 be a discrete random variable with probability mass function formula_19 depending on a parameter formula_1. Then the function
formula_21
considered as a function of formula_1, is the "likelihood function", given the outcome formula_3 of the random variable formula_4. Sometimes the probability of "the value formula_3 of formula_4 for the parameter value formula_1" is written as "P"("X" 
 "x" | "θ") or "P"("X" 
 "x"; "θ"). The likelihood is the probability that a particular outcome formula_3 is observed when the true value of the parameter is formula_1, equivalent to the probability mass on formula_3; it is "not" a probability density over the parameter formula_1. The likelihood, formula_32, should not be confused with formula_33, which is the posterior probability of formula_1 given the data formula_3.
Example.
Consider a simple statistical model of a coin flip: a single parameter formula_36 that expresses the "fairness" of the coin. The parameter is the probability that a coin lands heads up ("H") when tossed. formula_36 can take on any value within the range 0.0 to 1.0. For a perfectly fair coin, formula_38.
Imagine flipping a fair coin twice, and observing two heads in two tosses ("HH"). Assuming that each successive coin flip is i.i.d., then the probability of observing HH is
formula_39
Equivalently, the likelihood of observing "HH" assuming formula_38 is
formula_41
This is not the same as saying that formula_42, a conclusion which could only be reached via Bayes' theorem given knowledge about the marginal probabilities formula_43 and formula_44.
Now suppose that the coin is not a fair coin, but instead that formula_45. Then the probability of two heads on two flips is
formula_46
Hence
formula_47
More generally, for each value of formula_36, we can calculate the corresponding likelihood. The result of such calculations is displayed in Figure 1. The integral of formula_49 over [0, 1] is 1/3; likelihoods need not integrate or sum to one over the parameter space.
Continuous probability distribution.
Let formula_4 be a random variable following an absolutely continuous probability distribution with density function formula_51 (a function of formula_3) which depends on a parameter formula_1. Then the function
formula_54
considered as a function of formula_1, is the "likelihood function" (of formula_1, given the outcome formula_57). Again, formula_49 is not a probability density or mass function over formula_1, despite being a function of formula_1 given the observation formula_17.
Relationship between the likelihood and probability density functions.
The use of the probability density in specifying the likelihood function above is justified as follows. Given an observation formula_62, the likelihood for the interval formula_63, where formula_64 is a constant, is given by formula_65. Observe that
formula_66
since formula_67 is positive and constant. Because
formula_68
where formula_69 is the probability density function, it follows that
formula_70
The first fundamental theorem of calculus provides that
formula_71
Then
formula_72
Therefore,
formula_73
and so maximizing the probability density at formula_74 amounts to maximizing the likelihood of the specific observation formula_74.
In general.
In measure-theoretic probability theory, the density function is defined as the Radon–Nikodym derivative of the probability distribution relative to a common dominating measure. The likelihood function is this density interpreted as a function of the parameter, rather than the random variable. Thus, we can construct a likelihood function for any distribution, whether discrete, continuous, a mixture, or otherwise. (Likelihoods are comparable, e.g. for parameter estimation, only if they are Radon–Nikodym derivatives with respect to the same dominating measure.)
The above discussion of the likelihood for discrete random variables uses the counting measure, under which the probability density at any outcome equals the probability of that outcome.
Likelihoods for mixed continuous–discrete distributions.
The above can be extended in a simple way to allow consideration of distributions which contain both discrete and continuous components. Suppose that the distribution consists of a number of discrete probability masses formula_76 and a density formula_7, where the sum of all the formula_19's added to the integral of formula_51 is always one. Assuming that it is possible to distinguish an observation corresponding to one of the discrete probability masses from one which corresponds to the density component, the likelihood function for an observation from the continuous component can be dealt with in the manner shown above. For an observation from the discrete component, the likelihood function for an observation from the discrete component is simply
formula_80
where formula_81 is the index of the discrete probability mass corresponding to observation formula_3, because maximizing the probability mass (or probability) at formula_3 amounts to maximizing the likelihood of the specific observation.
The fact that the likelihood function can be defined in a way that includes contributions that are not commensurate (the density and the probability mass) arises from the way in which the likelihood function is defined up to a constant of proportionality, where this "constant" can change with the observation formula_3, but not with the parameter formula_1.
Regularity conditions.
In the context of parameter estimation, the likelihood function is usually assumed to obey certain conditions, known as regularity conditions. These conditions are assumed in various proofs involving likelihood functions, and need to be verified in each particular application. For maximum likelihood estimation, the existence of a global maximum of the likelihood function is of the utmost importance. By the extreme value theorem, it suffices that the likelihood function is continuous on a compact parameter space for the maximum likelihood estimator to exist. While the continuity assumption is usually met, the compactness assumption about the parameter space is often not, as the bounds of the true parameter values might be unknown. In that case, concavity of the likelihood function plays a key role.
More specifically, if the likelihood function is twice continuously differentiable on the k-dimensional parameter space formula_86 assumed to be an open connected subset of formula_87 there exists a unique maximum formula_88 if the matrix of second partials
formula_89 is negative definite for every formula_90 at which the gradient formula_91 vanishes,
and if the likelihood function approaches a constant on the boundary of the parameter space, formula_92 i.e.,
formula_93
which may include the points at infinity if formula_94 is unbounded. Mäkeläinen and co-authors prove this result using Morse theory while informally appealing to a mountain pass property. Mascarenhas restates their proof using the mountain pass theorem.
In the proofs of consistency and asymptotic normality of the maximum likelihood estimator, additional assumptions are made about the probability densities that form the basis of a particular likelihood function. These conditions were first established by Chanda. In particular, for almost all formula_3, and for all formula_96
formula_97
exist for all formula_98 in order to ensure the existence of a Taylor expansion. Second, for almost all formula_3 and for every formula_90 it must be that
formula_101
where formula_102 is such that formula_103 This boundedness of the derivatives is needed to allow for differentiation under the integral sign. And lastly, it is assumed that the information matrix,
formula_104
is positive definite and formula_105 is finite. This ensures that the score has a finite variance.
The above conditions are sufficient, but not necessary. That is, a model that does not meet these regularity conditions may or may not have a maximum likelihood estimator of the properties mentioned above. Further, in case of non-independently or non-identically distributed observations additional properties may need to be assumed.
In Bayesian statistics, almost identical regularity conditions are imposed on the likelihood function in order to proof asymptotic normality of the posterior probability, and therefore to justify a Laplace approximation of the posterior in large samples.
Likelihood ratio and relative likelihood.
Likelihood ratio.
A "likelihood ratio" is the ratio of any two specified likelihoods, frequently written as:
formula_106
The likelihood ratio is central to likelihoodist statistics: the "law of likelihood" states that degree to which data (considered as evidence) supports one parameter value versus another is measured by the likelihood ratio.
In frequentist inference, the likelihood ratio is the basis for a test statistic, the so-called likelihood-ratio test. By the Neyman–Pearson lemma, this is the most powerful test for comparing two simple hypotheses at a given significance level. Numerous other tests can be viewed as likelihood-ratio tests or approximations thereof. The asymptotic distribution of the log-likelihood ratio, considered as a test statistic, is given by Wilks' theorem.
The likelihood ratio is also of central importance in Bayesian inference, where it is known as the Bayes factor, and is used in Bayes' rule. Stated in terms of odds, Bayes' rule states that the "posterior" odds of two alternatives, &amp;NoBreak;&amp;NoBreak; and &amp;NoBreak;&amp;NoBreak;, given an event &amp;NoBreak;&amp;NoBreak;, is the "prior" odds, times the likelihood ratio. As an equation:
formula_107
The likelihood ratio is not directly used in AIC-based statistics. Instead, what is used is the relative likelihood of models (see below).
In evidence-based medicine, likelihood ratios are used in diagnostic testing to assess the value of performing a diagnostic test.
Relative likelihood function.
Since the actual value of the likelihood function depends on the sample, it is often convenient to work with a standardized measure. Suppose that the maximum likelihood estimate for the parameter θ is formula_108. Relative plausibilities of other θ values may be found by comparing the likelihoods of those other values with the likelihood of formula_108. The "relative likelihood" of θ is defined to be 
formula_110
Thus, the relative likelihood is the likelihood ratio (discussed above) with the fixed denominator formula_111. This corresponds to standardizing the likelihood to have a maximum of 1.
Likelihood region.
A "likelihood region" is the set of all values of θ whose relative likelihood is greater than or equal to a given threshold. In terms of percentages, a "p% likelihood region" for θ is defined to be
formula_112
If θ is a single real parameter, a p% likelihood region will usually comprise an interval of real values. If the region does comprise an interval, then it is called a "likelihood interval".
Likelihood intervals, and more generally likelihood regions, are used for interval estimation within likelihoodist statistics: they are similar to confidence intervals in frequentist statistics and credible intervals in Bayesian statistics. Likelihood intervals are interpreted directly in terms of relative likelihood, not in terms of coverage probability (frequentism) or posterior probability (Bayesianism).
Given a model, likelihood intervals can be compared to confidence intervals. If θ is a single real parameter, then under certain conditions, a 14.65% likelihood interval (about 1:7 likelihood) for θ will be the same as a 95% confidence interval (19/20 coverage probability). In a slightly different formulation suited to the use of log-likelihoods (see ), the test statistic is twice the difference in log-likelihoods and the probability distribution of the test statistic is approximately a chi-squared distribution with degrees-of-freedom (df) equal to the difference in df's between the two models (therefore, the e−2 likelihood interval is the same as the 0.954 confidence interval; assuming difference in df's to be 1).
Likelihoods that eliminate nuisance parameters.
In many cases, the likelihood is a function of more than one parameter but interest focuses on the estimation of only one, or at most a few of them, with the others being