Measure of similarity and diversity between sets
The Jaccard index is a statistic used for gauging the similarity and diversity of sample sets.
It is defined in general taking the ratio of two sizes (areas or volumes), the intersection size divided by the union size, also called intersection over union (IoU).
It was developed by Grove Karl Gilbert in 1884 as his ratio of verification (v) and now is often called the critical success index in meteorology. It was later developed independently by Paul Jaccard, originally giving the French name (community coefficient), and independently formulated again by T. Tanimoto. Thus, it is also called Tanimoto index or Tanimoto coefficient in some fields.
Overview.
The Jaccard index measures similarity between finite sample sets and is defined as the size of the intersection divided by the size of the union of the sample sets:
 formula_1
Note that by design, formula_2 If "A" intersection "B" is empty, then "J"("A", "B") = 0. The Jaccard index is widely used in computer science, ecology, genomics, and other sciences, where binary or binarized data are used. Both the exact solution and approximation methods are available for hypothesis testing with the Jaccard index.
Jaccard similarity also applies to bags, i.e., multisets. This has a similar formula, but the symbols used represent bag intersection and bag sum (not union). The maximum value is 1/2. 
 formula_1
The Jaccard distance, which measures "dis"similarity between sample sets, is complementary to the Jaccard index and is obtained by subtracting the Jaccard index from 1 or, equivalently, by dividing the difference of the sizes of the union and the intersection of two sets by the size of the union:
 formula_4
An alternative interpretation of the Jaccard distance is as the ratio of the size of the symmetric difference formula_5 to the union. 
Jaccard distance is commonly used to calculate an "n" × "n" matrix for clustering and multidimensional scaling of "n" sample sets.
This distance is a metric on the collection of all finite sets.
There is also a version of the Jaccard distance for measures, including probability measures. If formula_6 is a measure on a measurable space formula_7, then we define the Jaccard index by
 formula_8
and the Jaccard distance by
 formula_9
Care must be taken if formula_10 or formula_11, since these formulas are not well defined in these cases.
The MinHash min-wise independent permutations locality sensitive hashing scheme may be used to efficiently compute an accurate estimate of the Jaccard similarity index of pairs of sets, where each set is represented by a constant-sized signature derived from the minimum values of a hash function.
Similarity of asymmetric binary attributes.
Given two objects, "A" and "B", each with "n" binary attributes, the Jaccard index is a useful measure of the overlap that "A" and "B" share with their attributes. Each attribute of "A" and "B" can either be 0 or 1. The total number of each combination of attributes for both "A" and "B" are specified as follows:
formula_12 represents the total number of attributes where "A" and "B" both have a value of 1.
formula_13 represents the total number of attributes where the attribute of "A" is 0 and the attribute of "B" is 1.
formula_14 represents the total number of attributes where the attribute of "A" is 1 and the attribute of "B" is 0.
formula_15 represents the total number of attributes where "A" and "B" both have a value of 0.
Each attribute must fall into one of these four categories, meaning that
formula_16
The Jaccard similarity index, "J", is given as
formula_17
The Jaccard distance, "d""J", is given as
formula_18
Statistical inference can be made based on the Jaccard similarity index, and consequently related metrics. Given two sample sets "A" and "B" with "n" attributes, a statistical test can be conducted to see if an overlap is statistically significant. The exact solution is available, although computation can be costly as "n" increases. Estimation methods are available either by approximating a multinomial distribution or by bootstrapping.
Difference with the simple matching index (SMC).
When used for binary attributes, the Jaccard index is very similar to the simple matching coefficient. The main difference is that the SMC has the term formula_15 in its numerator and denominator, whereas the Jaccard index does not. Thus, the SMC counts both mutual presences (when an attribute is present in both sets) and mutual absence (when an attribute is absent in both sets) as matches and compares it to the total number of attributes in the universe, whereas the Jaccard index only counts mutual presence as matches and compares it to the number of attributes that have been chosen by at least one of the two sets.
In market basket analysis, for example, the basket of two consumers who we wish to compare might only contain a small fraction of all the available products in the store, so the SMC will usually return very high values of similarities even when the baskets bear very little resemblance, thus making the Jaccard index a more appropriate measure of similarity in that context. For example, consider a supermarket with 1000 products and two customers. The basket of the first customer contains salt and pepper and the basket of the second contains salt and sugar. In this scenario, the similarity between the two baskets as measured by the Jaccard index would be 1/3, but the similarity becomes 0.998 using the SMC.
In other contexts, where 0 and 1 carry equivalent information (symmetry), the SMC is a better measure of similarity. For example, vectors of demographic variables stored in dummy variables, such as gender, would be better compared with the SMC than with the Jaccard index since the impact of gender on similarity should be equal, independently of whether male is defined as a 0 and female as a 1 or the other way around. However, when we have symmetric dummy variables, one could replicate the behaviour of the SMC by splitting the dummies into two binary attributes (in this case, male and female), thus transforming them into asymmetric attributes, allowing the use of the Jaccard index without introducing any bias. The SMC remains, however, more computationally efficient in the case of symmetric dummy variables since it does not require adding extra dimensions.
Weighted Jaccard similarity and distance.
If formula_20 and formula_21 are two vectors with all real formula_22, then their Jaccard similarity index (also known then as Ruzicka similarity) is defined as
formula_23
and Jaccard distance (also known then as Soergel distance)
formula_24
With even more generality, if formula_25 and formula_26 are two non-negative measurable functions on a measurable space formula_7 with measure formula_6, then we can define
formula_29
where formula_30 and formula_31 are pointwise operators. Then Jaccard distance is
formula_32
Then, for example, for two measurable sets formula_33, we have formula_34 where formula_35 and formula_36 are the characteristic functions of the corresponding set.
Probability Jaccard similarity and distance.
The weighted Jaccard similarity described above generalizes the Jaccard Index to positive vectors, where a set corresponds to a binary vector given by the indicator function, i.e. formula_37. However, it does not generalize the Jaccard Index to probability distributions, where a set corresponds to a uniform probability distribution, i.e.
formula_38
It is always less if the sets differ in size. If formula_39, and formula_40 then
formula_41
Instead, a generalization that is continuous between probability distributions and their corresponding support sets is
formula_42
which is called the "Probability" Jaccard. It has the following bounds against the Weighted Jaccard on probability vectors.
formula_43
Here the upper bound is the (weighted) Sørensen–Dice coefficient.
The corresponding distance, formula_44, is a metric over probability distributions, and a pseudo-metric over non-negative vectors.
The Probability Jaccard Index has a geometric interpretation as the area of an intersection of simplices. Every point on a unit formula_45-simplex corresponds to a probability distribution on formula_46 elements, because the unit formula_45-simplex is the set of points in formula_46 dimensions that sum to 1. To derive the Probability Jaccard Index geometrically, represent a probability distribution as the unit simplex divided into sub simplices according to the mass of each item. If you overlay two distributions represented in this way on top of each other, and intersect the simplices corresponding to each item, the area that remains is equal to the Probability Jaccard Index of the distributions.
Optimality of the Probability Jaccard Index.
Consider the problem of constructing random variables such that they collide with each other as much as possible. That is, if formula_49 and formula_50, we would like to construct formula_7 and formula_52 to maximize formula_53. If we look at just two distributions formula_54 in isolation, the highest formula_53 we can achieve is given by formula_56 where formula_57 is the Total Variation distance. However, suppose we weren't just concerned with maximizing that particular pair, suppose we would like to maximize the collision probability of any arbitrary pair. One could construct an infinite number of random variables one for each distribution formula_58, and seek to maximize formula_53 for all pairs formula_54. In a fairly strong sense described below, the Probability Jaccard Index is an optimal way to align these random variables.
For any sampling method formula_61 and discrete distributions formula_54, if formula_63 then for some formula_64 where formula_65 and formula_66, either formula_67 or formula_68.
That is, no sampling method can achieve more collisions than formula_69 on one pair without achieving fewer collisions than formula_69 on another pair, where the reduced pair is more similar under formula_69 than the increased pair. This theorem is true for the Jaccard Index of sets (if interpreted as uniform distributions) and the probability Jaccard, but not of the weighted Jaccard. (The theorem uses the word "sampling method" to describe a joint distribution over all distributions on a space, because it derives from the use of weighted minhashing algorithms that achieve this as their collision probability.)
This theorem has a visual proof on three element distributions using the simplex representation.
Tanimoto similarity and distance.
Various forms of functions described as Tanimoto similarity and Tanimoto distance occur in the literature and on the Internet. Most of these are synonyms for Jaccard similarity and Jaccard distance, but some are mathematically different. Many sources cite an IBM Technical Report as the seminal reference.
In "A Computer Program for Classifying Plants", published in October 1960, a method of classification based on a similarity ratio, and a derived distance function, is given. It seems that this is the most authoritative source for the meaning of the terms "Tanimoto similarity" and "Tanimoto Distance". The similarity ratio is equivalent to Jaccard similarity, but the distance function is "not" the same as Jaccard distance.
Tanimoto's definitions of similarity and distance.
In that paper, a "similarity ratio" is given over bitmaps, where each bit of a fixed-size array represents the presence or absence of a characteristic in the plant being modelled. The definition of the ratio is the number of common bits, divided by the number of bits set ("i.e." nonzero) in either sample.
Presented in mathematical terms, if samples "X" and "Y" are bitmaps, formula_72 is the "i"th bit of "X", and formula_73 are bitwise "and", "or" operators respectively, then the similarity ratio formula_74 is
 formula_75
If each sample is modelled instead as a set of attributes, this value is equal to the Jaccard index of the two sets. Jaccard is not cited in the paper, and it seems likely that the authors were not aware of it.
Tanimoto goes on to define a "distance" based on this ratio, defined for bitmaps with non-zero similarity:
 formula_76
This coefficient is, deliberately, not a distance metric. It is chosen to allow the possibility of two specimens, which are quite different from each other, to both be similar to a third. It is easy to construct an example which disproves the property of triangle inequality.
Other definitions of Tanimoto distance.
Tanimoto distance is often referred to, erroneously, as a synonym for Jaccard distance formula_77. This function is a proper distance metric. "Tanimoto Distance" is often stated as being a proper distance metric, probably because of its confusion with Jaccard distance.
If Jaccard or Tanimoto similarity is expressed over a bit vector, then it can be written as
 formula_78
where the same calculation is expressed in terms of vector scalar product and magnitude. This representation relies on the fact that, for a bit vector (where the value of each dimension is either 0 or 1) then
formula_79
and
formula_80
This is a potentially confusing representation, because the function as expressed over vectors is more general, unless its domain is explicitly restricted. Properties of formula_81 do not necessarily extend to formula_25. In particular, the difference function formula_83 does not preserve triangle inequality, and is not therefore a proper distance metric, whereas formula_84 is.
There is a real danger