Comparing similarity between multivariate distributions I'm a non-mathematician trying to get a better intiution into how we can go about saying if one distribution is closer, or further away from another.
For example, if we have a set of patients who each have 20 blood test results and a multiclass label of diagnoses (A, B, C and D). How would one meaningfully say one class is closer or further away from another?
I imagine you can quantify this using KNN or 2-D PCA in seeing how their features (when reduced in dimension) relate to each other and have specific metrics which can provide a quantitation of similarity/difference?
In a similar fashion- would a model which performs well at differentiating between different classes given the features point towards there being is greater distance between classes?
Finally I've come across information theory and Kullback–Leibler divergence - presumably this is also useful in saying whether and how much one (multivariate) distribution compares with another?
Pointers on how to think about this much appreciated.


Finally I've come across information theory and Kullback–Leibler divergence - presumably this is also useful in saying whether and how much one (multivariate) distribution compares with another?

The KL divergence certainly can be a very useful metric to compare distributions because it (informally) defines a "distance", in terms of information, between one distribution, $p(x)$, and another distribution, $q(x)$, where the greater the distance, the more unique the two distributions are from each other. The use of "distance" also provides the notion that a KL divergence will always be non-negative, such that if two distributions are equal to each other, $p(x) = q(x)$, then the KL divergence will be 0.
In information theory, the KL divergence can be represented as the difference between the cross entropy of $p$ and $q$ and entropy of $p$:
$$D_{\mathbb{KL}}(p\,||\,q) = \mathbb{H}(p,q) - \mathbb{H}(p)$$
Entropy, and more generally information theory, deals in quantifying uncertainty.
Particularly, entropy, $\mathbb{H}(p)$, defines the innate uncertainty present in a random variable when sampled from, in this case, distribution $p$. If we are totally certain of the outcome for our random variable when sampled from distribution $p$, then its entropy is 0. The more uncertainty present however, the higher the entropy.
The cross entropy, $\mathbb{H}(p, q)$, defines how much more uncertainty there will be if we sample a random variable, with distribution $p$, when encoded to a new distribution $q$. The above representation of the KL divergence can be interpreted as finding that "excess" amount of uncertainty present in the cross entropy between distributions $p$ and $q$.
By this same logic (and to circle back to the top), if we had a hypothetical/magical dial that we could turn to reduce that "excess" information (kl divergence), we would be mutating distribution $q$ to be more alike distribution $p$.
