Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed. Some supervise the learning process by pairwise or tripletwise similarity constraints while others take advantage of structured similarity information among multiple data points. In this work, we approach deep metric learning from a novel perspective. We propose instance cross entropy (ICE) which measures the difference between an estimated instance-level matching distribution and its ground-truth one. ICE has three main appealing properties. Firstly, similar to categorical cross entropy (CCE), ICE has clear probabilistic interpretation and exploits structured semantic similarity information for learning supervision. Secondly, ICE is scalable to infinite training data as it learns on mini-batches iteratively and is independent of the training set size. Thirdly, motivated by our relative weight analysis, seamless sample reweighting is incorporated. It rescales samples' gradients to control the differentiation degree over training examples instead of truncating them by sample mining. In addition to its simplicity and intuitiveness, extensive experiments on three real-world benchmarks demonstrate the superiority of ICE.
Overview  
This paper proposes a new objective function called Instance Cross Entropy (ICE) for metric learning. Compared to the triplet loss (or contrastive loss) and its many variants, the distance between points in feature space is defined to be the dot/inner product, rather than computing the euclidian distance between feature vectors. Since the L2 norm of the final features is constrained  to be 1, the dot product represents the cosine distance between 2 feature vectors. Compared to the softmax/categorical loss, this objective has the advantage that there is no need to learn a per-class weight vector in the output softmax layer and therefore this method can be used even when the number of classes during training is unbounded. This is a very useful design feature, since the output softmax layer grows linearly with the size of the classes and can quickly become prohibitively large. Furthermore, mini batches for training can be randomly sampled without requiring expensive negative data mining strategies which are necessary when using the triplet loss. This is another very useful property. 
Overall, I think the ideas presented are interesting and the paper provides a useful summary of existing approaches for metric learning. However, I think the technical presentation needs to be improved significantly before  the paper can be accepted for publication. I found the paper quite difficult to follow. I had to re-read the motivation and the description of the loss function many times before being able to understand what is going on and how the proposed loss compares and contrasts with existing approaches. I also think some of the terminology and notation used in the paper is very confusing and should be updated to help the reader get to the main argument quickly. 
Comments
1. Title: What exactly does instance cross entropy (ICE) mean? And how is it different from categorical cross entropy (CCE)? Isn’t categorical cross-entropy also calculating the entropy between the predicted distribution for *each instance* and the ground truth? I think the authors should reconsider the name for the proposed loss and choose a more descriptive name for the algorithm.
2.  I also found the use of the term “matching distribution” a bit confusing since it has a well-defined meaning in statistics. However ML is full of overloaded terms and I understand if the authors want to keep this description. 
3. None of the references in the main text have brackets around them. I think this makes the paper appear very cluttered and makes parsing each paragraph quite difficult. I would highly recommend the paper be reformatted and the authors use brackets around references. 
4. In the abstract, the authors mention that the proposed method has a “clear probabilistic interpretation”. From my understanding, I see ICE as a blend of the categorical cross-entropy loss and the triplet loss, retaining useful properties of each without increasing the complexity of the loss computation. However, I do not see a clear probabilistic interpretation of the loss function. The softmax computation yields a probability distribution like output given a query and an anchor point, which is hand-designed in the objective function. I  think the authors should more clearly motivate what the probabilistic angle for the loss function is. 
5. I found Figure 1 to be very difficult to interpret. This is a missed opportunity, since this figure alone could communicate some of the key ideas in the paper to the reader. However, there are many details that have not been explicitly mentioned. What do the colours (white, blue, yellow) mean? The figure shapes probably mean classes. What do the terms p_* mean? What do i,j mean in Figure 1 e and how are they different from the previous figures. These details should all be present in the title for the figure. Even with these details, the figure could still be a bit more explicit. I found the yellow arrow with cross entropy and the notion of “ground truth” in the figure difficult to follow. 
6. How was the constrained optimisation performed? In Equation 6, we see that there is a constraint associated with each example in the mini batch which says the norm of the feature vector should be equal to 1. However I could not find any implementation details of how this constraint was satisfied. Did the authors use a Lagrangian formulation? I think the paper is irreproducible without this detail.  
Summary
I think this is an interesting proposal to combine the useful features of the softmax and triplet losses. However, I think the technical presentation needs to be improved significantly in order for the paper to be accepted for publication.