Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation
<|reference_start|>Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation: The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation. Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the 'Align-to-Distill' (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 De->Dsb and WMT-2014 En->De, respectively, compared to Transformer baselines.<|reference_end|>
@article{jin2024align-to-distill:,
    title={Align-to-Distill: Trainable Attention Alignment for Knowledge
  Distillation in Neural Machine Translation},
    author={Heegon Jin, Seonil Son, Jemin Park, Youngseok Kim, Hyungjong Noh,
  Yeonsoo Lee},
    journal={arXiv preprint arXiv:2403.01479},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.01479},
    primaryClass={cs.CL cs.AI}
}