Learning a Metric Embedding for Face Recognition using the Multibatch Method
which proves that the multibatch estimate is unbiased.
Next, we turn to analyze the variance of the multibatch estimate. let I ⇢ [k] 4 be all the indices i, j, s, t s.t. i 6 = j, s 6 = t, and we partition I to I 1 [ I 2 [ I 3 , where I 1 is the set where i = s and j = t, I 2 is when all indices are different, and I 3 is when i = s and j 6 = t or i 6 = s and j = t. Then:
For every r, denote by A (r) the matrix with A (r)
Observe that for every r,
Therefore,
Let us momentarily fix r and omit the superscript from A (r) . We consider the value of E ⇡ A ⇡(i),⇡(j) A ⇡(s),⇡(t) according to the value of q.
• For q = 1: we obtain E ⇡ A 2 ⇡(i),⇡(j) which is the variance of the random variable r r`i,j (z) r r L(z).
• For q = 2: When we fix i, j, s, t which are all different, and take expectation over ⇡, then all products of off-diagonal elements of A appear the same number of times in E ⇡ A ⇡(i),⇡(j) A ⇡(s),⇡(t) . Therefore, this quantity is proportional to P p6 =r v p v r , where v is the vector of all non-diagonal entries of A. Since P p v p = 0, we obtain (using Lemma 1) that P p6 =r v p v r  0, which means that the entire sum for this case is non-positive.
• For q = 3: Let us consider the case when i = s and j 6 = t, and the derivation for the case when i 6 = s and j = t is analogous. The expression we obtain is
. This is like first sampling a row and then sampling, without replacement, two indices from the row (while not allowing to take the diagonal element). So, we can rewrite the expression as:
where we denoteĀ i = E j6 =i A i,j and in the inequality we used again Lemma 1.
Finally, the bound on the variance follows by observing that the number of summands in I 1 is k 2 k and the number of summands in I 3 is O(k 3 ). This concludes our proof.
The paper addresses an important engineering challenge of developing a metric learning protocol which is computationally cheap while providing near state of the art performance. The task that the paper focuses on is face recognition.  The paper proposes a method to sample a dense set of loses to back prop. These loses are pairwise between k samples from different classes. The method is named multibatch, and the paper shows that the variance decreases as the square of the number of samples leading to faster convergence. The architecture proposed jointly aligns the faces and extracts a discriminative signature.  On the theoretical side, the paper presents an interesting view that metric learning can be seen as a harder problem. The multibatch method is a practical way to get SGD to converge quicker on the loss.
Major comments:   1) This reviewer's main concern is that although impressive performance on the LFW protocol is showcased in a very competitive setting, a atleast one or two more databases might be required to demonstrate that the protocol can scale up. Given that the practical contribution is primarily in terms of training through multibatch, more empirical evidence might be required. This reviewer imagines the results would generalize to other datasets, but given the strong focus of the paper on face recognition specifically, perhaps one or two more dataset results might be needed. This is the only major reason for the score of 2 on technical quality.  2) As most deep learning studies, this reviewer feels that perhaps the paper lacks a bit of intuition as to why a certain architecture worked. The final architecture employs a Network in network model, however given the fact that the paper aims to minimize model and computational complexity, it is a bit confusing (perhaps I m missing something) as to why the final model still seems more complex than a standard CNN. It would be more informative to know what other simpler models in fact did not work. I know that the paper mentions a VGG model that performed marginally better. This need not be in too much detail, however any discussion of that sort would provide more completeness and structure to the study.   Minor comments:  There are errors in the paper that have cropped up, eg. Fig 4 caption: LFW A thorough proofread is suggested.  Fig 2 is hard to read, could be made bigger  The experimental section could perhaps be given more structure, with small headings for each aspect of the experiment. It would improve readability as opposed to a big block of text.   Is the architecture shown in Table 2 in order of being applied to the input ? The exact ordering would be more informative than a simple listing for parameter count purposes.