text: For discrete probability distributions  and  defined on the same sample space,  the relative entropy from  to  is defined to be
  which is equivalent to
  In other words, it is the expectation of the logarithmic difference between the probabilities  and , where the expectation is taken using the probabilities .
  Relative entropy is only defined in this way if, for all ,  implies  (absolute continuity). Otherwise, it is often defined as  but the value  is possible even if  everywhere, provided that  is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.
  Whenever  is zero the contribution of the corresponding term is interpreted as zero because
  For distributions  and  of a continuous random variable, relative entropy is defined to be the integral
  where  and  denote the probability densities of  and .
  More generally, if  and  are probability measures on a measurable space  and  is absolutely continuous with respect to , then the relative entropy from  to  is defined as
  where  is the Radon–Nikodym derivative of  with respect to , i.e. the unique  almost everywhere defined function  on  such that  which exists because  is absolutely continuous with respect to . Also we assume the expression on the right-hand side exists. Equivalently (by the chain rule), this can be written as
  which is the entropy of  relative to . Continuing in this case, if  is any measure on  for which densities  and  with  and   exist (meaning that  and  are both absolutely continuous with respect to ), then the relative entropy from  to  is given as
  Note that such a measure  for which densities can be defined always exists, since one can take  although in practice it will usually be one that  in the context like counting measure for discrete distributions, or Lebesgue measure or a convenient variant thereof like Gaussian measure or the uniform measure on the sphere, Haar measure on a Lie group etc. for continuous distributions.   
  The logarithms in these formulae are usually taken to base 2 if information is measured in units of bits, or to base  if information is measured in nats. Most formulas involving relative entropy hold regardless of the base of the logarithm.
  Various conventions exist for referring to  in words. Often it is referred to as the divergence between  and , but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of  from  or as the divergence from  to . This reflects the asymmetry in Bayesian inference, which starts from a prior  and updates to the posterior . Another common way to refer to  is as the relative entropy of  with respect to  or the information gain from  over .