Discriminative Metric Learning by Neighborhood Gerrymandering
We formulate the problem of metric learning for k nearest neighbor classification as a large margin structured prediction problem, with a latent variable representing the choice of neighbors and the task loss directly corresponding to classification error. We describe an efficient algorithm for exact loss augmented inference, and a fast gradient descent algorithm for learning in this model. The objective drives the metric to establish neighborhood boundaries that benefit the true class labels for the training points. Our approach, reminiscent of gerrymandering (redrawing of political boundaries to provide advantage to certain parties), is more direct in its handling of optimizing classification accuracy than those previously proposed. In experiments on a variety of data sets our method is shown to achieve excellent results compared to current state of the art in metric learning.
The paper proposes a new approach to Mahalanobis metric learning which views metric learning as a structure-prediction problem. The structure we seek to establish is the set, $h$, of appropriate k-Nearest Neighbors (k-NNs) of an instance; appropriate in the sense that it will lead to a small classification error.  The idea of the authors is quite elegant: in learning a metric for k-NN prediction we can have a correct prediction for an instance $x_i$ as long as the majority of its k-NNs are of the same class as $x_i$. The metric learning algorithm they propose does exactly that, i.e. it looks for the Mahalanobis metric matrix $W$ that leads to k-nearest neighborhoods for each $x_i$ instance such that the majority of the instances of the neighborhood belongs to the correct class, i.e. that of $x_i$. Note here that there can be many such neighborhoods that predict the correct label for a given instance $x_i$, it is enough to prefer at least one of them over all the ones that predict the wrong label.  The authors define a structure prediction problem in which they optimize a tractable surrogate loss which is parametrized as a Mahalanobis distance in which for each instance the structure sought is the set of instances $h$ k-NNs that will have zero classification error. The authors build directly on the work of [12], McFee and Lankriet "Metric Learning to rank", ICML 2010, where metrics were learned to predict another type of structure, related to the k-NN, rankings.  They use the same feature map $\Psi(x,h)$ to quantify the relation of an instance $x$ to the $h$ structure (which is defined as the matrix that contains the negatives of the sums of the squared pairwise attribute differences of $x$ and its neighbors in $h$). This map under the Frobenious inner product with a PSD matrix $W$ leads to the sum of the negative Mahalanobis distances of $x$ from the instances of $h$, measuring in a sense the similarity of $x$ to $h$, S_W(x,h). The set of instances that maximizes S_W is the set of k-NNs of $x$, and the standard classification loss is the difference between the true label $x$ and the majority label of $h. Since this is intractable the authors define a tractable loss on the basis of the neighborhood that mostly violates the correct classication, using the sum of its loss (which plays the role of the margin) and its similarity to $x$, and the neighborhood that results in a correct classification and has the largest similarity to $x$.  The minimize the loss together with a frobenius norm regularizer on $W$. The minimization is done by stochastic gradient descent at every step of which they solve two neighborhood inference problems. One inference problem finds the neighborhood that has maximum similarity and classiciation-loss (loss augmented inference) and the other finds the maximum similarity zero classification-loss neighborhood.  The gradient update of the loss is then computed over these two inferenced values. The update rule learns a W such that it pulls instances close together if they belong in the zero classification loss neighborhood or far apart if the belong in a neighborhood that mostly violates the correct classication constraints, all other instances do not influence the updates. At each step there is a projection on the PSD matrix cone. In the experimental section the authors compare their method against LMNN, MLR (the metric learning to rank method), and the gradient boosted variant of LMNN on six datasets with small to medium numbers of instances, up to 20k and a few hundreds of features. The evaluated the performance of the different methods under k-NN, with k-NN=3,7,11. The empirical results show an advantage of their method over the baselines.  Overall I quite liked the idea presented in the paper and the definition of metric learning in which one has to find the right neighborhood, formulating the problem as a structure prediction problem. Even though it builds considerably on [12] I think that the optimization over the neighborhoods makes it substantially different from that paper, as well as interesting. One of the critisisms of the prior work and in particular LMNN is the fact that the target neighborhood structure remains fixed over the optimization. There are extensions of LMNN that do the update of the neighborhood as a part of learning, lifting that constraint, e.g. Jun Wang et al, Learning Neighborhoods for Metric Learning, ECML/PKDD 2012; the authors should discuss them and ideally include them as a baseline in the experiments since they also learn the neighborhood.  I am missing a discussion on the computational complexity especically in view of the small to medium size of the datasets used in the evaluation. How fast does the method converge, what are the problem sizes to which it scales. Also the experimental section is a bit weak, the hyper-parameters of the different methods as well as the grids from which they take values are not described fully. In addition I would have welcomed a more extensive methods with other metric methods which have been shown to have good predictive performance, for example Jun Wang et al, Parametric Local Metric Learning for Nearest Neighbor Classification, in NIPS 2012, which has been shown to have better performance than the baseline methods against which the authors compare.