Sequence-to-sequence models such as transformers, which are now being used in a wide variety of NLP tasks, typically need to have very high capacity in order to perform well. Unfortunately, in production, memory size and inference speed are all strictly constrained. To address this problem, Knowledge Distillation (KD), a technique to train small models to mimic larger pre-trained models, has drawn lots of attention. The KD approach basically attempts to maximize recall, i.e., ranking Top-k" tokens in teacher models as higher as possible, however, whereas precision is more important for sequence generation because of exposure bias. Motivated by this, we develop Knowledge Acquisition (KA) where student models receive log q(y t |y <t , x) as rewards when producing the next token y t given previous tokens y <t and the source sentence x. We demonstrate the effectiveness of our approach on WMT'17 De-En and IWSLT'15 Th-En translation tasks, with experimental results showing that our approach gains +0.7-1.1 BLEU score compared to token-level knowledge distillation.
[Overview]
In this paper, the authors proposed a new method called knowledge acquisition (KA) for distilling the learned knowledge from the teacher model to the student model. Unlike the conventional KL-divergence based knowledge distillation method, the authors take the reverse version which learns the student to increase the precision. The paper gave a thorgough analysis on the proposed KA strategy and compared with other strategies like KL and JS divergence. On the sequence generation task (translation), the authors showed that the proposed KA strategy achieved better performance compared with KD based methods when distilling the knowledge from a teacher model to a student model.
[Pros]
1. The authors proposed a new strategy to perform the knowledge distillation from a teacher model to student model for sequence generator. To improve the precision of the student model, the authors proposed to invert the formula of KL divergence, i.e., the position of prediction probability from teacher and student models.
2. The authors presented a thorough analysis on the proposed KA strategy and compared it with KL strategy in terms of the precision and recall for the student models. I think it is very readable and understandable. This analysis align with those put on generative adversarial network.
3. The authors performed the experiments on the translation tasks showing that the proposed KA strategy outperforms both KL and JS strategy in terms of the generation performance. Also, the authors ablated the number of top reference tokens from the teacher model and showed that using a reasonable number of top tokens is important to help alleviate the noised in the teacher model.
[Cons]
The main concern about the proposed method is whether it can be used as a generic strategy for transferring the knowledge from the teacher model to student model.
1. First, I have a doubt on the stability of the proposed strategy. In my opinion, the improvements on the sequence generation tasks are mainly due to the tuned hyper parameters for the training, especially the lambda in Eq(12), which is tuned at the validation set. It controls how much to modulate the prediction distribution of student model toward that of teacher model. However, a less tuned lambda would cause either over curve fitting or under curve fitting. As a result, the authors should: 1) first show how the performance would be affected by varying the lambda in the formula; 2) from the reading, I did not see whether the lambda was tuned as well for KD or (KD + KA) / 2. If not, then for fair comparison, the authors should tune the lambda for the KD and (KD + KA) / 2 strategy as well. At some point, I would think the combination of KD and KA would be better than either of them.
2. Second, some experimental results are somewhat counter-intuitive to me. These are two folds: a) In Figure 4(b), we can see that the KA strategy has learned to generate more new tokens compared with KD. This is a bit strange to me because, KA will focus on the precision instead of recall. To me, pushing the precision will high likely sacrifice the recall and thus the number of novel tokens generated by the model. b) similarly, in Figure 5, it is shown that KA has generally higher entropy than KD. This is also a bit counter intuitive. In Eq(6), it is obvious that the proposed strategy has a entropy term which will be reduced when we want to reduce the KL divergence during the training time. From Figure 5, KA and KD start from the same point (I guess it is because the same pre-trained student model) is used. However, for KA, the entropy start to increase and then converge to a stable number which is consistently higher than KD strategy.
3. Third, Figure 4(a) also indicates some thing. When only the top few tokens are used to transfer the knowledge from teacher model to student model, KA focus on the precision of a small subspace, which tends to have few modes. However, when the number of tokens is increased, the mode number would also increase drastically. i guess that’s why the both strategies finally become very close to each other, and the minor gap between them is probably due to the benefit from hyper-parameter fine-tuning.
Besides the above comments. there are some minor points which are missed in the paper:
1. As pointed above, it is not clear whether the same tuning is also applied to KD and (KD + KA) / 2. the authors should mention this in the experiment section.
2. It is also not clear how many tokens is used for reporting the numbers in Table 2. Is it the whole vocabulary side? If this is the case, the gap between KA and KD on validation set are pretty close while more significant on test set.
3. In Eq (11), should there be a minus sign before the expectation?
4. Also, is there any more comment on why it is hard to train the student model joint from scratch? what will happen in this case?
[Summary]
In this paper, the authors proposed a new strategy called Knowledge Acquisition which is used for distilling the knowledge learned from  teacher model to the student model. Different from KD strategy, it inverted the position of probability distributions for teacher and student models. By this way, the KA strategy learns a student model which can achieves higher precision. The proposed strategy is evaluated on sequence generation, particularly translation task.  However, as pointed above, in my opinion, there are some counter-intutive observations in the experimental results. It would be good if the authors can address these concerns in the rebuttal.