Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples. The problem is generally solved by learning an embedding, such that the samples of the same category are close (compact) while samples from different categories are far away (spread-out) in the embedding space. One popular way of generating such embeddings is to use the second-to-last layer of a deep neural network trained as a classifier with the softmax cross-entropy loss. In this paper, we show that training classifiers with different temperatures of the softmax function lead to different distributions of the embedding space. And finding a balance between the compactness, "spread-out" and the generalization ability of the feature is critical in metric learning. Leveraging these insights, we propose a "heating-up" strategy to train a classifier with increasing temperatures. Extensive experiments show that the proposed method achieves state-of-the-art performances on a variety of metric learning benchmarks.
Summary:
This paper proposes a novel optimization strategy regarding softmax cross-entropy loss, to extract the effective features of well generalization in the framework of metric learning.
The authors focus on the "temperature" parameter in the softmax and through analyzing the role of the temperature in terms of gradient, propose the approach of heating-up softmax in which the temperature is varied from low to high in training.
And, the effects of normalization such as by l2 and BatchNorm are discussed in the framework of heated-up softmax.
The experimental results on metric learning tasks demonstrate the effectiveness of the proposed method in comparison with the other methods.
Comments:
Pros:
+ The idea of heating up the temperature in softmax is interesting, and seems novel in the literature of metric learning.
+ The performance improvement, especially produced by batchNorm-based normalization, is shown.
Cons:
- The formulation of tempered softmax with normalization is already presented in [Wang et al., 2017].
- The reason why the heating-up approach contributes to better metric learning is not clearly provided in a well convincing way.
- It lacks an important ablation study to fairly validate the method.
- The discussion/comparison is limited to the simple softmax function.
Although the reviewer likes the idea of heating up softmax, this paper can be judged as a borderline slightly leaning toward reject, due to the above weak points, the details of which are explained as follows.
- Formulation
The softmax equipped with temperature for the normalized features and weights are shown in [Wang et al., 2017]. The only difference from that work is the way to deal with temperature; in [Wang et al., 2017], the temperature is "optimized" as a trainable parameter, while it is dealt with in a hand-crafted way of heating up in this work. Honestly speaking, it is unclear which approach is better, though the optimization in [Wang et al., 2017] seems elegant as stated in that paper. The only way to validate this work compared to [Wang et al., 2017] is to empirically evaluate those two methods in the experiments. Such a comparison experiment is not found and it is a main flaw of this paper.
- Justification of the method
The gradients of the softmax cross-entropy loss parameterized with a temperature T are well analyzed in Sections 3.1&3.2. But, in Section 3.3, the reviewer cannot find the clear and convincing explanation for why the temperature T should be increased in the training. My question is: why don't you use alpha=4 consistently throughout the training?
 It might be related to the process of simulated annealing (though "temperature" is usually cooled down in SA), and more interestingly, it would also be possible to find connection with the work of [Guo et al., 2017]. In [Guo et al., 2017], the temperature in the softmax is optimized as a post processing for calibrating the classifier outputs. Though the calibration task itself is a little bit apart from the metric learning of the authors' interest, we can find in that paper an interesting result that the temperature is heated up to increase the confidence of the classifier outputs, which is quite similar to the process of fine-tuning by heating up softmax as done in this work. Therefore, the reviewer guesses that the effectiveness of heating up softmax can also be interpreted from the viewpoint of [Guo et al., 2017].
There is also less description about Figure 1; in particular, the reviewer cannot understand what Figure 1(d) means.
- Ablation study
To empirically resolve the above concerns, it is necessary to present the empirical comparison with the "static" softmax.
Namely, the methods of HLN/HBN should be carefully compared to LN/BN of "alpha=4", not only those of alpha=16 shown in Table 1&2; the comparison in Table 3 seems unfair since the authors apply the static softmax without normalization.
And, it would be better to show the performance of heated-up softmax "without" normalization to show the important role of the normalization, as done in [Wang et al., 2017].
In summary, since the proposed method is composed of a heating-up approach and feature normalization, the authors are required to validate the method from those two aspects, respectively, for increasing the significance of this paper.
- Other loss function
For achieving a compactness in feature representation, the simple softmax requires both temperature and normalization. It, however, is also conceivable to employ the other types of loss function for that purpose, such as [a] which is based on the (Mahalanobis) distance with taking into account the margin between categories. The distance based loss also embeds features into localized clusters, which satisfies the authors' objective in this work. To validate the proposed method, it is required to compare the method with such a different types of loss function.
[a] Wan, W., Zhong, Y., Li, T., & Chen, J. (2018). Rethinking Feature Distribution for Loss Functions in Image Classification, In CVPR2018, pp. 9117–9126.