Comparing similarity between multivariate distributions I'm a non-mathematician trying to get a better intiution into how we can go about saying if one distribution is closer, or further away from another.
For example, if we have a set of patients who each have 20 blood test results and a multiclass label of diagnoses (A, B, C and D). How would one meaningfully say one class is closer or further away from another?
I imagine you can quantify this using KNN or 2-D PCA in seeing how their features (when reduced in dimension) relate to each other and have specific metrics which can provide a quantitation of similarity/difference?
In a similar fashion- would a model which performs well at differentiating between different classes given the features point towards there being is greater distance between classes?
Finally I've come across information theory and Kullback–Leibler divergence - presumably this is also useful in saying whether and how much one (multivariate) distribution compares with another?
Pointers on how to think about this much appreciated.

To elaborate on whuber's comment, consider the case of only categorial features, which means that the distribution is a frequency vector $\vec{h}=(h_1,\ldots,h_d)$. You can compute the Euclidean distance as suggested (your proposal of a 2D PCA projection implicitly uses Euclidean distance, because PCA approximately preserves Euclidean distances), i.e.
$$d(\vec{h},\vec{g})=\sqrt{\sum_{i=1}^d (h_i-g_i)^2}=\sqrt{\langle\vec{h}-\vec{g},\vec{h}-\vec{g}\rangle}=\sqrt{(\vec{h}-\vec{g})^T(\vec{h}-\vec{g})}$$
This does, however, not take into account the distances between the levels that the components of the vector represent. Consider, e.g., the case of a color image where the components represent cells in a color space: In this case, the distance between the cell midpoints should be taken into account (how to possibly define it, is a can of worms that I do not wnat to open here). One way to take it into account is by introducing an "interaction matrix" $A$
$$d(h,g)=\sqrt{(\vec{h}-\vec{g})^T A(\vec{h}-\vec{g})}$$
While it might be possible to estimate $A$ from training data (e.g. $A=\Sigma^{-1}$, which leads to the Mahalanobis distance), it is generally better to construct $A$ on basis of domain knowledge. The same holds for other distance measures between histograms like the "earth mover's distance" which requires a (domain specific) distance between the cells.
