  While non-autoregressive (NAR) models are showing great promise for machine
translation, their use is limited by their dependence on knowledge distillation
from autoregressive models. To address this issue, we seek to understand why
distillation is so effective. Prior work suggests that distilled training data
is less complex than manual translations. Based on experiments with the
Levenshtein Transformer and the Mask-Predict NAR models on the WMT14
German-English task, this paper shows that different types of complexity have
different impacts: while reducing lexical diversity and decreasing reordering
complexity both help NAR learn better alignment between source and target, and
thus improve translation quality, lexical diversity is the main reason why
distillation increases model confidence, which affects the calibration of
different NAR models differently.

Computation and Language