Learning Deep Embeddings with Histogram Loss
We suggest a loss for learning deep embeddings. The new loss does not introduce parameters that need to be tuned and results in very good embeddings across a range of datasets and problems. The loss is computed by estimating two distribution of similarities for positive (matching) and negative (non-matching) sample pairs, and then computing the probability of a positive pair to have a lower similarity score than a negative pair based on the estimated similarity distributions. We show that such operations can be performed in a simple and piecewise-differentiable manner using 1D histograms with soft assignment operations. This makes the proposed loss suitable for learning deep embeddings using stochastic optimization. In the experiments, the new loss performs favourably compared to recently proposed alternatives.
The authors provide a new loss function for learning embeddings in deep networks, called histogram loss. This loss is based on a pairwise classification: whether two labels belong to the same class or not. In particular, the authors suggest to look at the similarity distribution of the embeddings on the L2 unit sphere (all embeddings are L2 normalized). The idea is to look at the distribution of the similar embedding (positive pairs) and the distribution of the non-similar ones (negative pairs) and make the probability that positive pairs  has smaller score then negative pairs, smaller.  After reviewing previous work in the area (Section 2), in Section 3 they develop a method how to estimate the Histogram loss. They begin with the definitions of Histogram (eq. 1, and 2) and based on them give the definition of $p_reverse$ which later we they want to minimize (eq.3). Afterward, they present an approximation to $p_reverse$ and show that it is differentiable. They conclude  Section 3 with showing a connection to the quadruplet loss.  In Section 4 the authors compare their method to other state of the art embedding loss, specifically, they consider the binomial deviance loss (eq. 7 and 8) and Lifted Structured Similarity Softmax (LSSS; eq. 9). They try these loses on 4 datasets: CUB200-2011 [25], CUHK03 [11], Market-1501 [29] and Online Products.   In addition, different architectures for the neural network are used: For training on the CUB-200-2011 and the Online Products they use GoogleNet while for CUHK03 and Market-1501 they used the Deep Metric Learning architecture. The optimizatin algorithm used in this work is Adam (for stochastic optimization).  They show the results using the Recall@K performance on all the datasets with different methods in Figures 3 and 4. The authors show how their method is sometime competitive and sometimes outperforms previous state of the art algorithms in the embeddings area.   
I like and see the benefit of the authors' method for embeddings. The (almost) parameter free method is very appealing and have its benefits. The authors simply show the results of their algorithm and demonstrate that it works. Also, after many works where I had to read the supplementary material in order to understand the work, I appreciate the authors non-supplementary paper! Also, their throughout comparison to other methods in terms of loss functions, datasets, and different parameters, is very good. Also, I like the choice of the datasets, where there are many classes, which seems natural to this kind of loss.   Two things bother me in this work:  1) The authors do not explain what is recall@K, where I needed to go back through the papers "Deep Metric Learning via Lifted Structured Feature Embedding" and then "Product quantization for nearest neighbor search". I could postulate what is the definition in this case is, but still it affected the self-contained nature of this work.  2) After doing so much profound work, it was not clear to me why each dataset has different set of graphs. I miss a good reason why such comparison is adequate in this context, especially in the light of that the authors did most of the work for throughout comparison.   Typos: * In figure 5 caption, should it read |Red is for *positive* pair, green is for *negative* pairs?