Sample Complexity of Learning Mahalanobis Distance Metrics
Metric learning seeks a transformation of the feature space that enhances prediction quality for a given task. In this work we provide PAC-style sample complexity rates for supervised metric learning. We give matching lower-and upper-bounds showing that sample complexity scales with the representation dimension when no assumptions are made about the underlying data distribution. In addition, by leveraging the structure of the data distribution, we provide rates fine-tuned to a specific notion of the intrinsic complexity of a given dataset, allowing us to relax the dependence on representation dimension. We show both theoretically and empirically that augmenting the metric learning optimization criterion with a simple norm-based regularization is important and can help adapt to a dataset's intrinsic complexity yielding better generalization, thus partly explaining the empirical success of similar regularizations reported in previous works.
Summary In this article the authors present a theoretical analysis of non-regularized supervised metric learning formulations. They study two frameworks. On the one hand, the distance-based framework considers algorithms that optimize a distance w.r.t. class label information. On the other hand, the classifier-based framework considers algorithms which optimize a metric for a specific prediction or a ranking task.For both frameworks, the authors provide PAC style bounds (sample complexity bounds) on the difference between the true risk and the empirical risk of the learned metric. They also analyze the difference between the true risk of the learned metric and the true risk of the best in-class metric and discuss the the necessity of a strong dependency on the dimensionality of the data. In the last part of the paper, the authors consider some regularized formulation using the Frobenius norm as aregularizer, they propose some refinement of their results in this context and argue that adding a regularization term provides better guarantees. The authors propose finally an empirical evaluation of two well known metric learning approaches (namely ITML and LMNN) showing that adding a regularization term is indeed beneficial. The paper is relatively clear and well written. The results are new and provide new theoretical understanding for metric learning . Comments.-Correct the proof of Lemma 2 to make it independent from the dimension. You should add some comments to make a link between Th1 and Th2. In the current form of the paper, they are presented separately and with formulations that do not allow the reader to directly get the consequences on the dependence on D.-Mention that the results of Jin et al. (NIPS 2009) can be extended to replace the dependency on d by a term based on the data concentration (term B in author's paper), for any distributions and convex regularized formulations.This can be done by using the techniques of Bousquet & Elisseeff, JMLR 2002 - note that the convexity of the formulation is mandatory for applying algorithmic stability. +Bellet et al. Metric Learning. Morgan & Claypool publishers. 2015. (chapter 8)-One of the claim of the paper is to generalize some previous results to possibly non convex loss. However, in this context there is no guarantee to find the best model (matrix) minimizing the risk which makes the results a bit non informative on what can be done in practise.Do not forget that even for convex formulation, taking into all the quadratic constraints involved by classical metric learning formulations is heavy and in general some heuristics are used to reduce the number of constraints. One interesting perspective would be to know if the dependency on D impacts more severely non convex non regularized formulations.-The authors have focused on the dependence over the representation dimension D, however it could be interesting to consider also the true rank of the matrix M rather than the data dimension. This would allow to make some relationships with metric learning based on Cholesky decomposition of the learned metric and methods making use of some low rank regularization. This point should be at least discussed in the perspectives.-Considering, the classifier-based approach, the bibliography lacks a reference on methods learning a linear classifier from good similarity functions, even if this paper adresses other settings. +Balcan et al. Improved Guarantees for Learning via Similarity Functions. COLT 2008. +Bellet et al. Similarity Learning for Provably Accurate Sparse Linear Classification. ICML 2012. This aspect is also studied in the paper of Guo and Ying cited by the authors.Overall, the results are interesting and I think that providing theoretical guarantees on classifiers making use of the result of a known metric is very important and results in that direction must continued.The results are based on covering and fat shattering dimensions arguments, is there any perspective to use other frameworks (Rademacher-based for example?)-About the experiments.Even though they do not consider noisy data in their analysis, the authors create noise augmented datasets to see the influcence of the regularization. The conclusion of the experiments is that regularization can help to deal with noisy datasets or in other words they point out that the well known problem of overfitting can be partly adressed by choosing a good regularizer which is known fact. The scaling for the x-axis is different for the 3 figures, what is the reason for that? In particular, one can wonder what happens after 120/150 for Iris and Ionosphere? For the comparison with ITML, it could be interesting to have the accuracy obtained by directly using the rank-1 matrix alone in the distance to compare the influence of the regularizer. It seems that the choice of this matrix may do a lot of the "job". The experiments lack of datasets - 3 is very small and the number of dimensions is relatively low - and lack of methods for comparison. We can also wonder why no experiment on any classifier-based setting has been made. There also exist other datasets with higher dimensions in the UCI datasets, you can also check here : +Y. Shi et al.:Sparse Compositional Metric Learning. AAAI 2014: 2078-2084. The experiments would have gained in interest, if they authors have presented the evolution of the bounds with the data and with the dimension.  Other comments:-line 075: the authors consider binary classification, however the loss considered below allows to consider a multiclass setting and some data sets in the experiments are multiclass. -I am wondering why in the definition of the true risk of the classifier-based approach, the margin is chosen at 1/2, while for the empirical risk the margin can be fixed to any value between 0 and 1/2. -line 219: "do now know" -> "do not know" -in the proof of Lemma 4, the notion of covering is used extensively. Is it possible to extend the results to other regularization norms?