Learning group representation is a commonly concerned issue in tasks where the basic unit is a group, set or sequence. Previously, the computer vision community tries to tackle it by aggregating the elements in a group based on an indicator either defined by human such as the quality or saliency of an element, or generated by a black box such as the attention score or output of a RNN. This article provides a more essential and explicable view. We claim the most significant indicator to show whether the group representation can be benefited from an element is not the quality, or an inexplicable score, but the discrimiability. Our key insight is to explicitly design the discrimiability using embedded class centroids on a proxy set, and show the discrimiability distribution w.r.t. the element space can be distilled by a light-weight auxiliary distillation network (DDNet). This procedure is denoted as discriminability distillation learning (DDL). We show the proposed DDL can be flexibly plugged into many group based recognition tasks without influencing the training procedure of the original tasks. Comprehensive experiments on set-to-set face recognition and action recognition valid the advantage of DDL on both accuracy and efficiency, and it pushes forward the state-of-the-art results on these tasks by an impressive margin.
This paper proposes a discriminability indicator using embedded class centroids on a proxy set, and show the discriminability distribution w.r.t. the element space distilled by a light-weight auxiliary distillation network, which is called discriminability distillation learning (DDL).
This methodology was tested on a selection of datasets addressing set-to-set face recognition and action recognition. I have the following concerns, which made me to suggest rejecting this paper.
1) The methodology explained in Section 3 should be improved. In the way it is presented, it is not clear and reproducible. In detail,
1.1)  It is better to describe the pipeline together with Figure 1.
1.2) Contribution of the top part of Figure 1 in proposed pipeline is not clear. More descriptive caption should be added.
1.3) In section 1, where you ask reader to observe the feature embedding of elements which lie close to the centroid… Which section or figure are you referring to?
1.4) In subsection 2.3, clearly discuss how proposed approach is different than traditional distillation methods. Also, explain your modifications and discuss why new structure is better than default ResNet-18. 
1.5) Subsections 3.2-3.4 do not contain enough explanation to support equation of proposed method. 
1.6) It is not clear why authors use cosine distance, because it is simple? 
1.7) The quality scores generated by DDL is very hard to interpret, which to me the one of the biggest problem of the proposed methodology.
2) The related work section should include studies more related to the motivation behind DDL (“….to select represent samples from the whole set efficiently for group understanding.”). For instance, attention models, saliency detection, key-frame detection methods, outlier detection methods, quality aware networks, etc. should be discussed and compared with DDL. Instead, authors more focused on face recognition and action recognition tasks themselves. Therefore, many related works are missing. Especially, Section 2.3., should not be a part of related work but should be integrated either to Section 1 (Introduction) or Section 3.
3) Experimental analysis: 
3.1 ) It is not possible to figure out if the proposed method is really performing better than SOA. In other words, the experimental analysis should be extended. One thing authors should consider is including more datasets, especially more challenging datasets i.e., the ones not already saturated. Some examples can be: iLIDS-VID, PRID2011, YouTube Face, IJB-A benchmark.
3.2) Is there any reason to mainly focus on face recognition and action recognition?
3.3) “For face recognition, DDL can easily improve the performance by concentrating the discriminative information and for video action recognition, DDL can further accelerate the pipeline by eliminating the frames with insufficient information.” This sounds like DDL behaves differently depends on the task, but perhaps the conclusion driven by the authors cannot be fully correct given that the number of used datasets are limited to make such a general statement. I suggest authors to re-write this sentence.
3.4) The experimental analysis should include comparisons with methodologies such as quality learning via attention mechanism, etc. to better understand the necessity and effectiveness of proposed DDL. However, only for one dataset such a comparison was performed. It is strange why for other datasets the same comparison was not done.
3.5) Experimental analyses so much focuses on to justifying the performance improvements of DDL is independent to the base model selected (e.g. mainly Table 3,  but also Table 4, 5 and 6 include). However, to me this is a supporting experiment and it is more important to show the real necessity of the proposed method by comparing it with SOTA methods.
3.6) I do not find using Youtube Face benchmark suitable as it is already saturated, i.e. many existing models perform around 97%, which does not allow to show whether DDL can significantly contribute to the task or not. From the corresponding results, it looks like DDL does not provide any significant improvement.
3.7) In Table 3, it is hard to observe the contribution of DDL in terms of the performance. The improvements are in the level of 1%. More discussion is needed about that table.
3.8) “The discriminability distillation learning is more practical to untrimmed video action recognition since there are more diversity videos chip with ambiguous content and visual blur problem”  This means that DDL should not be used in some tasks, it is not generalizable, task-specific? Please elaborate it more clearly.
3.9) Table 2 is hard to interpret, improve the discussions regarding it.
3.10) Table 1 and table 2 were not referred in the text. What does DATA refer to in Table 1, why they are different?, if they are really different i.e. it is not me who misinterpret, then is not it unfair to compare the results?
4) Other suggestions:
4.1) Support your statements with references and appropriate experimental analysis. 
4.2) In Section 4, only YTF dataset is mentioned but authors also use IQIYI-VID-FACE challenge dataset.
4.4) Section 4.2. is not clear at all, and should be re-written.
4.5) Some references to the tables are wrong or some are missing, pls. check them all. E.g., in section 4.2., paragraph two, table 3 should be table 2.
4.6) There is no reference to the any figures in the text.
4.7) Paper contains several typos and grammatical mistakes. Also, most of the points were not explained well.