Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.
 This paper proposed to transfer the classifier from the model for face classification to the task of alignment and verification. The problem setting is interesting and valuable, however, the contribution is not clearly demonstrated. Specifically, it proposed to utilize the teacher model from classification to other tasks, and proposed a unified objective function to model the transferability as shown in Equation (5). The two terms in (5), (7) and (9) are used to transfer the knowledge from the teacher model. It maybe possible to claim that the different terms may play different roles for different tasks. However, there should be some general guidelines for choosing these different terms for regularization, rather than just make the claim purely based on the final results. In table 4 and table 5, the results seem to be not so consistent for using the distillation loss. The author mentioned that it is due to the weak teacher model. However, the teacher model just differs in performance with around 3% in accuracy. How could we define the “good” or “bad” of a teacher model for model distillation/transfer? Besides, it seems that the improvement comes largely from the trick of initialization as mentioned in Section 3.2. Hence, it is still not clear which parts contribute to the final performance improvements. It could be better if the authors can report the results from each of the components together. The authors just try the parameter (.alpha, .beta) to be (0,0), (1,0), (0,1) and (1,1). I think the range for both values could be any positive real value, and how about the performance for other sets of combinations, like (0.5, 0.5)?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposed to transfer the classifier from the model for face classification to the task of alignment and verification. The problem setting is interesting and valuable, however, the contribution is not clearly demonstrated. Specifically, it proposed to utilize the teacher model from classification to other tasks, and proposed a unified objective function to model the transferability as shown in Equation (5). The two terms in (5), (7) and (9) are used to transfer the knowledge from the teacher model. It maybe possible to claim that the different terms may play different roles for different tasks. However, there should be some general guidelines for choosing these different terms for regularization, rather than just make the claim purely based on the final results. In table 4 and table 5, the results seem to be not so consistent for using the distillation loss. The author mentioned that it is due to the weak teacher model. However, the teacher model just differs in performance with around 3% in accuracy. How could we define the “good” or “bad” of a teacher model for model distillation/transfer? Besides, it seems that the improvement comes largely from the trick of initialization as mentioned in Section 3.2. Hence, it is still not clear which parts contribute to the final performance improvements. It could be better if the authors can report the results from each of the components together. The authors just try the parameter (.alpha, .beta) to be (0,0), (1,0), (0,1) and (1,1). I think the range for both values could be any positive real value, and how about the performance for other sets of combinations, like (0.5, 0.5)?
assistant
 This paper proposed to transfer the classifier from the model for face classification to the task of alignment and verification. The problem setting is interesting and valuable, however, the contribution is not clearly demonstrated. Specifically, it proposed to utilize the teacher model from classification to other tasks, and proposed a unified objective function to model the transferability as shown in Equation (5). The two terms in (5), (7) and (9) are used to transfer the knowledge from the teacher model. It maybe possible to claim that the different terms may play different roles for different tasks. However, there should be some general guidelines for choosing these different terms for regularization, rather than just make the claim purely based on the final results. In table 4 and table 5, the results seem to be not so consistent for using the distillation loss. The author mentioned that it is due to the weak teacher model. However, the teacher model just differs in performance with around 3% in accuracy. How could we define the “good” or “bad” of a teacher model for model distillation/transfer? Besides, it seems that the improvement comes largely from the trick of initialization as mentioned in Section 3.2. Hence, it is still not clear which parts contribute to the final performance improvements. It could be better if the authors can report the results from each of the components together. The authors just try the parameter (.alpha, .beta) to be (0,0), (1,0), (0,1) and (1,1). I think the range for both values could be any positive real value, and how about the performance for other sets of combinations, like (0.5, 0.5)?