A measure of difference between two points
In mathematics, specifically statistics and information geometry, a Bregman divergence or Bregman distance is a measure of difference between two points, defined in terms of a strictly convex function; they form an important class of divergences. When the points are interpreted as probability distributions – notably as either values of the parameter of a parametric model or as a data set of observed values – the resulting distance is a statistical distance. The most basic Bregman divergence is the squared Euclidean distance.
Bregman divergences are similar to metrics, but satisfy neither the triangle inequality (ever) nor symmetry (in general). However, they satisfy a generalization of the Pythagorean theorem, and in information geometry the corresponding statistical manifold is interpreted as a (dually) flat manifold. This allows many techniques of optimization theory to be generalized to Bregman divergences, geometrically as generalizations of least squares.
Bregman divergences are named after Russian mathematician Lev M. Bregman, who introduced the concept in 1967.
Definition.
Let formula_1 be a continuously-differentiable, strictly convex function defined on a convex set formula_2.
The Bregman distance associated with "F" for points formula_3 is the difference between the value of "F" at point "p" and the value of the first-order Taylor expansion of "F" around point "q" evaluated at point "p":
formula_4
formula_20
formula_25
Here, formula_26 and formula_27 are the dual points corresponding to p and q.
Moreover, using the same notations :
formula_28
Properties.
For any formula_45
formula_46
formula_48
formula_52. Then 
For any formula_56,
formula_57
This is an equality if formula_58 is in the relative interior of formula_51.
In particular, this always happens when formula_51 is an affine set.
Proofs.
Fix formula_65 . Take affine transform on formula_66 , so that formula_67.
Take some formula_68, such that formula_69. Then consider the "radial-directional" derivative of formula_66 on the Euclidean sphere formula_71.
formula_72 for all formula_73.
Since formula_74 is compact, it achieves minimal value formula_75 at some formula_76.
Since formula_66 is strictly convex, formula_78. Then formula_79.
Since formula_80 is formula_81 in formula_82, formula_83 is continuous in formula_82, thus formula_40 is closed if formula_34 is.
Fix formula_89. Take some formula_90 , then let formula_91. Then draw the Bregman ball formula_92. It is closed and bounded, thus compact. Since formula_93 is continuous and strictly convex on it, and bounded below by formula_94, it achieves a unique minimum on it.
By cosine law, formula_95, which must be formula_96, since formula_58 minimizes formula_93 in formula_51, and formula_51 is convex.
If formula_103, then since formula_104 is in the relative interior, we can move from formula_58 in the direction opposite of formula_104, to decrease formula_107 , contradiction.
Thus formula_108.
Classification theorems.
 ;templatestyles src="Math_proof/styles.css" / ;Proof
For any formula_112 , define formula_113, the set of all probability measures on formula_114, with formula_115.
Define a divergence on formula_116 as any function of type formula_117, such that formula_118 for all formula_119, then:
Given a Bregman divergence formula_124, its "opposite", defined by formula_125, is generally not a Bregman divergence. For example, the Kullback-Leiber divergence is both a Bregman divergence and an f-divergence. Its reverse is also an f-divergence, but by the above characterization, the reverse KL divergence cannot be a Bregman divergence.
formula_133 
is generated by the negative entropy function 
formula_134 
When restricted to the simplex, the last two terms cancel, giving the usual Kullback–Leibler divergence for distributions.
formula_135
is generated by the convex function
formula_136
Generalizing projective duality.
A key tool in computational geometry is the idea of projective duality, which maps points to hyperplanes and vice versa, while preserving incidence and above-below relationships. There are numerous analytical forms of the projective dual: one common form maps the point formula_137 to the hyperplane formula_138. This mapping can be interpreted (identifying the hyperplane with its normal) as the convex conjugate mapping that takes the point p to its dual point formula_26, where "F" defines the "d"-dimensional paraboloid formula_140.
If we now replace the paraboloid by an arbitrary convex function, we obtain a different dual mapping that retains the incidence and above-below properties of the standard projective dual. This implies that natural dual concepts in computational geometry like Voronoi diagrams and Delaunay triangulations retain their meaning in distance spaces defined by an arbitrary Bregman divergence. Thus, algorithms from "normal" geometry extend directly to these spaces (Boissonnat, Nielsen and Nock, 2010)
Generalization of Bregman divergences.
Bregman divergences can be interpreted as limit cases of skewed Jensen divergences (see Nielsen and Boltz, 2011). Jensen divergences can be generalized using comparative convexity, and limit cases of these skewed Jensen divergences generalizations yields generalized Bregman divergence (see Nielsen and Nock, 2017).
The Bregman chord divergence is obtained by taking a chord instead of a tangent line.
Bregman divergence on other objects.
Bregman divergences can also be defined between matrices, between functions, and between measures (distributions). Bregman divergences between matrices include the Stein's loss and von Neumann entropy. Bregman divergences between functions include total squared error, relative entropy, and squared bias; see the references by Frigyik et al. below for definitions and properties. Similarly Bregman divergences have also been defined over sets, through a submodular set function which is known as the discrete analog of a convex function. The submodular Bregman divergences subsume a number of discrete distance measures, like the Hamming distance, precision and recall, mutual information and some other set based distance measures (see Iyer &amp; Bilmes, 2012 for more details and properties of the submodular Bregman.)
For a list of common matrix Bregman divergences, see Table 15.1 in.
Applications.
In machine learning, Bregman divergences are used to calculate the bi-tempered logistic loss, performing better than the softmax function with noisy datasets.
Bregman divergence is used in the formulation of mirror descent, which includes optimization algorithms used in machine learning such as gradient descent and the hedge algorithm.
References.
 ;templatestyles src="Reflist/styles.css" / ;
 ;templatestyles src="Refbegin/styles.css" / ;