Recently, parallel text generation has received widespread attention due to its success in generation efficiency. Although many advanced techniques are proposed to improve its generation quality, they still need the help of an autoregressive model for training to overcome the one-to-many multi-modal phenomenon in the dataset, limiting their applications. In this paper, we propose latent -GLAT, which employs the discrete latent variables to capture word categorical information and invoke an advanced curriculum learning technique, alleviating the multi-modality problem. Experiment results show that our method outperforms strong baselines without the help of an autoregressive model, which further broadens the application scenarios of the parallel decoding paradigm. 1 Introduction Non-autoregressive Transformer (NAT, Gu et al., 2018) introduce a parallel decoding paradigm with higher decoding efficiency (> 10  ) than autoregressive models (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Unlike autoregressive models, NAT models impose conditional independence assumptions in words to support parallel decoding of sentences during inference. It attracts many researchers to explore NAT in machine translation (Gu et al., 2018; Lee et al., 2018; Kaiser et al., 2018) and text-to-speech tasks (Chen et al., 2019; Peng et al., 2020). Amount of researchers devoted themselves to improve the NATs' inferior generation quality. Such as modeling word inter-dependencies by curriculum learning (Guo et al., 2020a; Liu et al., 2020) or iterative refinements mechanism (Ghazvininejad * Shujian Huang is the corresponding author.  Work is done while at ByteDance AI Lab.  The implementation of latentGLAT will be released at https://github.com/baoy-nlp/Latent-GLAT . et al., 2019; Guo et al., 2020b), introducing latent variables to decompose target sentences and serve as the springboard for decoding (Shu et al., 2019; Ma et al., 2019; Bao et al., 2021), and introduce inductive bias for models' training (Wei et al., 2019; Li et al., 2019). The most successful method is the glancing transformer (GLAT, Qian et al., 2021a), which trains the NAT model by sampling partial target words as inputs to predict the remaining target words, explicitly building dependencies between the observed and unobserved words. Qian et al. (2021b) employ GLAT to achieve impressive results on the translation task of WMT21 1 , even outperforming many strong autoregressive translation systems in BLEU score (Papineni et al., 2002). Although existing NAT models achieve competitive results compared to autoregressive models in translation tasks, it is not negligible that they still need the help of an autoregressive Transformer (AT, Vaswani et al., 2017) as a teacher for training, i.e., sequence-level knowledge distillation (Kim and Rush, 2016). A well-recognized explanation is a multi-modality problem (Zhou et al., 2020; Sun and Yang, 2020): each input may have multiple valid outputs in datasets, which will prevent NAT models from learning to organize consistent outputs. Training with the outputs of an AT can directly bypass the multi-modal phenomenon in the dataset, effectively improving the models' performances. However, training NAT models by knowledge distillation are limited. First, it needs to train an extra AT model, which inevitably enlarges the training cost. Second, it is hard to promise that the teacher (or AT) model can be accurate enough in all text generation settings, which will become the bottleneck for its student NAT model. Therefore, training a model from scratch without the help of an AT model is still an open and interesting problem. the multi-modality problem following a divide-and-conquer spirit, introducing a small set of discrete latent variables to capture the target word categorical information and divide the origin goal into latent variables modeling and sentence reconstruc-tion. First, the categorical information may have fewer multi-modality phenomena than the original words, thus can be learned directly without the help of knowledge distillation. Second, the word categorical information is informativeness to the sentence reconstruction. We can extend glancing training with these discrete latent variables for modeling the sentence, encouraging the model to build dependencies on word categorical information rather than words, which works more robustly. Experiment results on WMT14, Quora, and DailyDialog datasets show that latentGLAT achieves remarkable improvements over several strong baselines, verifying the effectiveness of latentGLAT. More impressively, latentGLAT even outperforms autoregressive models in Quora and DailyDialog datasets, further validating our motivation for removing knowledge distillation. In-depth analyses indicate that the introduced discrete latent variables are helpful to alleviate the multi-modality problem and are necessary for performance improvement. For a sequence-to-sequence task of predicting sequence Y = ( y 1 , y 2 ,    , y m ) given its input sequence X = ( x 1 , x 2 ,    , x n ) , the classical autoregressively factorization decomposes the p ( Y | X ) with a series of conditional probability: <t 1 2 t  1 Although such factorization achieved great success in previous studies (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017), they predict each word 2 based on the prefix words, which may suffer from the issues of error accumulation and slow decoding during inference. the above problems, Gu et al. (2018) firstly propose non-autoregressive Transformer (NAT), introduc-2 introduc-2 We use BPE segmentation in our experiments, and they are strictly tokens. For clarity, we use words and tokens interchangeably in the paper. ing a non-autoregressive factorization as: p NAT ( Y | X ) = m (cid:89) t =1 p ( y t | X ) , (2) where each word y t are modeled independently. During inference, the NAT model can decode the word simultaneously by arg max y t p ( y t | X ) for each y t , remarkably improving the efficiency (15  speedups to an autoregressive Transformer). However, the independence assumption may prevent the NAT model from leveraging the inherent word dependencies to organize consistent outputs. Due to this,the efficiency improvements of NAT are at the cost of its quality, e.g., the performance degradation by more than 10.0 BLEU (Papineni et al., 2002) points in machine translation tasks (Gu et al., 2018). Besides, recent studies (Zhou et al., 2020; Sun and Yang, 2020) point out that the multimodality phenomenon in the dataset aggravates the challenge of NAT models. Glancing Transformer. To mitigate the issue of missing word dependency in NAT models, Qian et al. (2021a) propose Glancing Transformer (GLAT), introducing glancing training (GLT) and sampling partial target tokens for training NAT: LGLAT =  log p ( Y obs | Y obs , X ) =  (cid:88) y i  Y obs log p ( y i | Y obs , X ) , (3) where Y obs is the partial target tokens, and Y obs is its complements set. It progressively decreases the sampling ratio and obtains better performances in machine translation tasks. Nevertheless, we find that GLAT in experiments still has a multi-modality problem 3 : First, its sampling rate cannot be decreased to zero during training, which exists the issue of exposure bias . Second, it still heavily relies on a teacher model for further improvements (Qian et al., 2021a). Latent Transformer. To alleviate the multimodality problem, Kaiser et al. (2018); Shu et al. (2019); Ma et al. (2019); Bao et al. (2021) propose Latent Transformer (LT), introducing latent variables z for NAT predictions as: p LT ( Y | X ) = (cid:90) z p ( z | X )  p ( Y | z , X ) . where p LT ( Y | X ) is always trained by variational inference (Ma et al., 2019) or discretization techniques (Kaiser et al., 2018). Such latent variables are decomposed from the target sentence, which is informative to determine the mode of the sentence and alleviates the multi-modality problems. Although Latent Transformer models improve performance in terms of BLEU score, their used autoregressive predictor (Kaiser et al., 2018; Bao et al., 2021) or deep iterative transformation (Shu et al., 2019; Ma et al., 2019) for predicting latent variables unavoidable sacrifice the overall decoding efficiency. Besides, they do not explicitly build the interdependencies among the outputs. In this section, we present latentGLAT. latent-GLAT follows Latent Transformer models (Kaiser et al., 2018; Bao et al., 2021) but introduces glancing training (Qian et al., 2021a) with the discrete latent variables. Our intuitions are as follows: First, compared to the words, the introduced discrete latent variables may have fewer modes than words and be informative to determine the modes of the sentences. In such a case, we can directly learn the discrete latent variables by the Glancing Transformer (Qian et al., 2021a), keeping competitive inference efficiency. More importantly, we can employ the latent variables to invoke glancing training for modeling the target sentences, which is informative enough to reduce the multi-modality problem of original sentences. Besides, glancing at latent variables also works robustly due we can obtain the latent variables during inference. In this part, we state the structure of latentGLAT, which introduces a small set of discrete latent variables for a NAT model, basically following Kaiser et al. (2018); Roy et al. (2018); Bao et al. (2021). Let K be the size of the discrete latent space and let [ K ] denote the set { 1 , 2 ,    , K } . For each target sentence Y = ( y 1 , y 2 ,    , y m ) , we use a same-length latent variable sequence for modeling it as: p ( Y | X ) = (cid:88) z p  ( z | X )  m (cid:89) t =1 p  ( y t | z , X ) , (5) where z = ( z 1 , z 2 ,    , z m ) and z i  [ K ] ,  is the model parameters. Discretization. For discretizing target sentences to latent variables, we use vector quantization (Roy et al., 2018), which works by dividing a large set of origin vector representations into small groups. We assign each token y i with a group j  [ K ] that has the nearest distance to its representation: z i = arg min j  [ K ] || repr( y i )  q j || 2 , (6) where q  RK  d model is the maintained representations and d model is its dimension. We use the embedding as repr( y i ) , refer to Bao et al. (2021). Finally, the model is trained to minimize LLT = LLP + LWP , (7) where LWP and LLP are the prediction loss for words Y and latent variables z , respectively. The maintained representations q are updated with an exponential moving average over a mini-batch of target tokens { y 1 ,    , y i ,    } : c j  c j + (1   ) (cid:88) i 1 [ z i = j ] , q j   q j + (1   ) (cid:88) i 1 [ z i = j ] repr( y i ) c j (8) where c j is assigned count for group j , and we set decay parameter  = 0 . 999 in our experiments. Encoder), a latent predictor FLP (NAT Predictor), and a decoder FDEC (Mix. Decoder). We parameterize them with the multi-head attention-based encoder or decoder, similar to Transformer (Vaswani et al., 2017). Their functions can be formalized as: ( e 1 , e 2 ,    , e n )  FENC ( x 1 , x 2 ,    , x n ) , ( h 1 , h 2 ,    , h m )  softcopy( e 1: n ) , p  ( z | X )  FLP ( h 1: m , e 1: n ) , p  ( Y | z , X )  FDEC ( z 1: m , h 1: m , e 1: n ) , where we use an extra module FLEN to predict the target length m and initialize the decoder inputs H = ( h 1 , h 2 ,    , h m ) with the softcopy (Wei et al., 2019) mechanism. As shown in Figure 2b, we eventually employ words to invoke glancing training for minimizing LWP , namely we optimize the FDEC by minimizing LGLTWP =  log p  ( Y obs | z obs , Y obs , X ) , (11) where Y obs and z obs are the sampled target tokens and discrete latent variables. We find Eqn. (10) works robustly in experiments and analyze it in Section (  4.3). The small number ( K < 128 ) of discrete latent variables can capture high-level categorical information of the target words, supporting better learning design for parallel sequence decoding. Our first insight is that we can learn to non-autoregressively predict the discretized latent variables directly without the help of distillation. Specifically, we parameterize the FLP in a non-autoregressive fashion and use a glancing training technique (GLT, Qian et al., 2021a) for optimizing it, as shown in Figure 2a: LGLTLP =  log p  ( z obs | z obs , X ) (9) where z obs is uniformly sampled from z , refer to Qian et al. (2021a). We provide more training details of latentGLAT in Appendix B. Our next insight is modeling the sentence based on the sampled latent variables z obs rather than z , namely, glancing at z obs for optimizing FDEC : LWP =  log p  ( Y | z obs , X ) . Overall Training Loss. Our full-fledged loss includes latent variable prediction, sentence recon-struction, and length prediction losses: L = LGLTWP + LGLTLP +  LLEN , (12) where  = 0 . length, latent variables, and sentence in turn. For the target length, latentGLAT first predicts the target length m with the length predictor FLEN . To avoid the length prediction errors during inference, latentGLAT expands the length m to a ranges (we use [ m  3 ,    , m + 2] , total six candidates in our experiments). Then, latentGLAT predicts the latent variables  z with arg max z p  ( z | X ) and sentence  Y with arg max Y p  ( Y |  z , X ) for each candidate. Similar to Ma et al. (2019), latentGLAT also ranks the candidates by itself ( self-reranking ) and chooses the highest score output with:  Y = arg max Y p  ( Y |  z , X )   | Y | (13) where  is the length penalty ratio to avoid the length bias, and | Y | denotes the length of Y . We conduct experiments on several generation tasks, including machine translation, paraphrase generation, and dialog generation. Dataset. We chose the most popular benchmarks for each task:  Machine Translation (MT) : We follow previous practices in NAT models and use the WMT14 English (EN)  German (DE) corpus (4.5M sentence pairs) and the IWSLT14 German (DE)  English (EN) corpus (160K sentence pairs) to validate our proposed model. We obtain the datasets following the instruction open-sourced in fairseq 4 . In detail, we first tokenize the datasets with Moses script. Then, we use 37,000 and 10,000 operations to split the words into byte-pair encodings (BPE, Sennrich et al., 2016) in WMT14 and IWSLT14 datasets, respectively. We also share subword embeddings between the source and target language for each dataset. Paraphrase Generation (PG) : We use the Quora 5 dataset to evaluate the paraphrase generation task. The Quora dataset contains around 135K labeled paraphrases pairs. Following the standard dataset split, we sample 100K sentence pairs from the labeled paraphrases as training data and hold out 30K pairs for testing, the remaining about 5K pairs for validation. Like the MT tasks, we tokenize the corpus with Moses scripts and split the words into BPE units with total 32K operations. Dialog Generation (DG) : We conduct the dialog generation experiments on the DailyDialog dataset (Li et al., 2017). We obtain the processed DailyDialog dataset from Bao et al. (2020) 6 . The training set contains 87,170 sentence pairs (11,118 dialogues). The validation and testing set in the dataset contain 8069 pairs (1000 dialogues) and 7740 pairs (1000 dialogues), respectively. Note that these tasks emphasize different aspects. The task of MT aims to transfer bilingual sentences with semantically invariant conditions. The PG task differs from machine translation and works on 4 https://github.com/pytorch/fairseq 5 https://www.kaggle.com/c/ quora-question-pairs/data 6 https://github.com/gmftbyGMFTBY/ MultiTurnDialogZoo mode transformation in the same language, whose goal is to synthesize a sentence different from the original input but conveys the same meaning. The DG task is most challenging due to the complex generation goal. Implementations. We compare latentGLAT with Transformer (Vaswani et al., 2017), NAT (Gu et al., 2018), and GLAT (Qian et al., 2021a) models. We implement them based on the open-source framework fairseq (Ott et al., 2019). For machine translation tasks, we use the base setting ( d model = 512 , d hidden = 2048 , dropout = 0 . 1 , n head = 8 , and n layer = 6 ) of Transformer (Vaswani et al., 2017) for WMT14 dataset and a smaller setting ( d model = 512 , d hidden = 1024 , dropout = 0 . 3 , n head = 4 , and n layer = 6 ) for IWSLT14 dataset. The number of layers in latentGLAT decoder and latent predictor are both set to 4 in experiments. We use inverse square root learning rate scheduling for WMT14 and a linear annealing learning rate from 3 . 0  10  4 to 1 . 0  10  5 in 250K steps for IWSLT14. The models are optimized with Adam (Kingma and Ba, 2015) optimizer (  1 = 0 . 9 ,  2 = 0 . 999 ) in 300K steps for WMT14 and 250K steps for IWSLT14. As for the ratio  that used in glancing sampling, we linear anneal the ratio from 0 . 5 to 0 . 3 in whole training steps. The mini-batch in each step consists of 2K tokens for IWSLT14 and 64K tokens for WMT14. Since the scale of the Quora and DailyDialog datasets are close to the IWSLT14, we keep the same setting to the IWSLT14, such as the Adam, learning rate (linear annealing from 3 . 0  10  4 to 1 . 0  10  5 ), and batch size (2K tokens). Evaluation. To validate the effectiveness of our proposed method, we evaluate it in terms of quality and efficiency. We use tokenized and cased BLEU scores (Papineni et al., 2002) 7 to evaluate the generation quality of MT and PG tasks. For dialog generation, we also include BLEU-1 and BLEU-2 scores for analysis. Following the common practices (Gu et al., 2018; Qian et al., 2021a), we measure the decoding latency of each model by decoding sentence by sentence and compute the speedup compared with the autoregressive Transformer (AT) model to reflect its decoding efficiency. We highlight the best NAT result. We can see from Table 1 that our latentGLAT almost outperforms all the NAT baselines (NAT and GLAT) in generation quality on all tasks while keeping a competitive decoding speedup to the autoregressive counterpart. Machine Translation. As seen, without the help of an AT model for training, the vanilla NAT and advanced GLAT model only obtain inferior generation quality. In contrast, latentGLAT achieves competitive generation quality in machine translation tasks, indicating that the introduced latent variables effectively reduce the multimodality issue and support glancing training well. It narrows the performance gap between non-autoregressive decoding and autoregressive decoding from 11.46 (GLAT vs. AT) to 2.34 ( latent-GLAT vs. AT) BLEU points on WMT14 EN  DE task while keeping a high-speed decoding efficiency. Paraphrasing. Unlike the translation task, the performance gap between non-autoregressive and autoregressive decoding on the paraphrase generation task is minor (NAT vs. AT,  3 . 32 BLEU points, GLAT vs. AT,  0 . 96 BLEU points ). Nevertheless, introducing discrete latent variables still is helpful to obtain a better performance. latent-GLAT realizes a non-autoregressive model with better performance than the autoregressive model on Quora ( latentGLAT vs. AT, +1 . 14 points). Dialog Generation. We can see a different trend on the DailyDialog dataset  an AT model performs poorly than NAT models. Both GLAT and latentGLAT outperform the AT model in BLEU-1, BLEU-2, and BLEU scores, indicating that these models recall more reference tokens and organize the tokens well. We conjecture that the weak and indirect association between the inputs and outputs of the dialogue Models WMT14 IWSLT14 Speedups  EN  DE DE  EN DE  EN CMLM 1  10.88 -CMLM 4  22.06 - 9.79  CMLM 10  24.65 - 3.77  LevT 2 . 05 24.43 -2.93  LV-NAR 11.80 -22.30  SynST 20.74 25.50 23.82 4.86  Flowseq 20.85 25.40  1.10  CNAT 21.30 25.73 29.81 10.37  AT 27.17 31.53 34.29 1.00  NAT 10.78 15.19 17.77 15.29  GLAT 16.71 24.78 29.07 15.29  latentGLAT 24.71 29.16 32.31 11.31  Table 2: BLEU scores and speedups of different models trained with raw datasets on machine translation tasks. results in this unusual phenomenon. Specifically, the weak connection may encourage the AT model to predict the tokens by paying more attention to their history outputs, which degenerate to a target-side language model. In contrast, the NAT models do not have this fast track, pushing them to pay more attention to the inputs and recall more target tokens. We further find that there are so-called safe response (Li et al., 2016) in AT's outputs, which verify our conjecture. More Comparisons. we further compare the advanced NAT models that builds upon latent variables or iterative refinement in machine translation tasks:  NATs w/ latent variables: LV-NAR (Shu et al., 2019), SynST (Akoury et al., 2019), Flowseq (Ma et al., 2019), and CNAT (Bao et al., 2021). Iterative NATs: CMLM (Ghazvininejad et al., 2019) and LevT (Gu et al., 2019). Table 2 shows that introducing latent variables 8403 Figure 3: BLEU scores and their relative decoding speedups of different models on WMT14 EN  DE test set. (LV-NAR, Flowseq, and CNAT) or decoding with multiple iterations (CMLM and LevT) both improve non-autoregressive decoding in translation quality. However, iterative refinements or deep transformations always sacrifice decoding efficiency. In contrast, the proposed latentGLAT outperforms all NAT models with a relatively low cost, keeping a competitive speedup over autoregressive Transformer (AT). Specifically, latentGLAT with one-pass decoding narrows the performance gap to the AT from 5.87 BLEU points to 2.34 BLEU points on the WMT14 EN  DE test set. Decoding efficiency. We can see there is a tradeoff between the translation quality and decoding efficiency in Table 2. We thus present the scatter plot of different models in Figure 3, showing the trend of translation quality and decoding efficiency. As seen, latentGLAT is located on the top-right of the baselines. It outperforms the baselines in the BLEU score if decoding speedup is fixed and in decoding speedup if the BLEU score is fixed. We now turn to verify our intuition that latent-GLAT can alleviate the multi-modality problem. latentGLAT largely alleviates the sentence-level multi-modal problem. Previous researches (Gu et al., 2018; Ma et al., 2019; Qian et al., 2021a; Bao et al., 2021) always utilize a Transformer model as a teacher for training NAT models, namely sequence-level knowledge distillation (Kim and Rush, 2016), which can Methods WMT14 IWSLT14 Avg   EN  DE DE  EN DE  EN NAT 10.78 15.19 17.77 +6.58 w/ KD 17.69 22.02 23.78 GLAT 16.71 24.78 29.07 +5.19 w/ KD 25.21 29.84 31.07 Flowseq 20.85 25.40 24.75 +2.87 w/ KD 23.72 28.39 27.55 CNAT 21.30 25.73 29.81 +3.08 w/ KD 25.56 29.36 31.15 latentGLAT 24.71 29.16 32.31 +0.95 w/ KD 26.64 29.93 32.47 Table 3: BLEU scores of NAT models trained with (or without) knowledge distillation (KD) on translation tasks. directly reduces the sentence-level multi-modal phenomenon in datasets. Therefore, we use the average gains from the knowledge distillation to reflect the ability of the NAT models to overcome this issue. As seen in Table 3, the pure NAT models heavily rely on knowledge distillation. By introducing the target information with the latent variables (Flowseq and CNAT) or sampled tokens (GLAT), the NAT models improve its' ability to overcome the multi-modality issue. Our proposed latent-GLAT well combines the above two techniques. It obtains only 0.95 BLEU points average gains and validates our motivation. raw sentences. To validate our intuition that the introduced latent variables are easier to predict than tokens, we refer to Zhou et al. (2020) to compute the complexity metrics on each dataset according to alignment relations. Specifically, we use the fast_align 8 toolkit to align source input X and target outputs Y or discretized latent variable se-8 https://github.com/clab/fast_align 8404 L# Introduce z Glancing Training BLEU (  )  with z with Y 1 12.60 2 (cid:88) 13.43 (+0.83) 3 (cid:88) 17.11 (+4.51) 4 (cid:88) (cid:88) 18.88 (+6.20) 5 (cid:88) (cid:88) 22.35 (+9.75) 6 (cid:88) (cid:88) (cid:88) 23.64 (+11.04) Table 5: BLEU scores of different latentGLAT con-figurations on the WMT14 EN  DE valid set. quences z . Then, we compute the token-level complexity CTOK ( d ) and the sentence-level complexity CSEN ( d ) according to Zhou et al. (2020). These metrics can trivially understand as the number of valid candidates for each input. As shown in Table 4, the latent variables have the lowest complexity in both token-level complexity and sentence-level complexity. In other words, predicting the latent variable sequences is effortless than predicting others, which is consistent with our intuition. Although we obtain a lower complexity dataset by filtering the datasets with an autoregressive model (AT outputs versus Raw outputs), they may introduce model error and need extra training for AT model. In contrast, the discrete latent variables are simple and informative enough to serve as a springboard for modeling target sentences. performance with a large margin. We can see in Table 5 that introducing latent variables both obtain performance gains to their counterpart (L#2 vs. L#1, +0 . 83 points, and L#4 vs. L#3, +1 . 69 points). As expected, the gains are largely improved while adopting the glancing training with discrete latent variables (L#5 vs. L#1, +9 . 75 points), which already outperforms glancing training with the reference token (L#5 vs. L#4, +3 . 55 points). Finally, we jointly perform glancing training with the reference tokens and discrete latent variables, achieving the best result (L#6 vs. L#1, +11 . 04 points). Effects of K and  . As shown in Figure 4 and Table 6, we search the hyper-parameter of latent-GLAT that the number of discrete latent variables and the length penalty ratio  according to the validation performance. We notice that using more latent codes causes performance degradation during inference, in which the latent variables may degenerate to tokens and contains more prediction error during inference. The latentGLAT implemented with 64 latent variables and  = 1 . 1 obtains the best result on WMT14 EN  DE valid set. Gu et al. (2018) first propose a non-autoregressive Transformer (NAT) model for neural machine translation (NMT) and begin to explore parallel decoding. It abandons explicitly modeling word interdependencies to decode the tokens in parallel, sig-nificantly improving the inference speed. However, its translation quality is inferior to the Transformer (Vaswani et al., 2017). To alleviate this performance degradation, many researchers work to enhance word dependency modeling, including imitation learning (Wei et al., 2019; Li et al., 2019), curriculum learning (Guo et al., 2020a; Liu et al., 2020), iterative refinements (Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019; Guo et al., 2020b; Huang et al., 2022), and a simplified autoregressive process (Sun et al., 2019). The most representative method is the glancing transformer model (Qian et al., 2021a), which adaptively and progressively samples partial tokens as inputs and predicts the remaining tokens, effectively establishing the dependencies between the sampled tokens and the remaining tokens. However, these models still rely on a teacher 8405 for training, which cannot directly learn the raw dataset that contains one-to-many multi-modality phenomenon. Introducing latent variables (Bao et al., 2019, 2021) to organize the target sentence is also a helpful route. Among them, our method is close to Kaiser et al. (2018); Shu et al. (2019); Ma et al. (2019); Akoury et al. (2019); Bao et al. (2021). These methods decompose the latent variables (hints) from the target sentence and divide the origin goal into two parts: modeling latent variables and modeling the target sentences based on latent variables. It implicitly overcomes the multimodality phenomenon of target sentences because the latent variables can largely determine the mode of the sentence. However, these methods always model the latent variables with an autoregressive predictor, which naturally sacrifices the decoding efficiency. Unlike them, our approach models the discrete latent variables in a non-autoregressive fashion and extends glancing training with the discrete latent variables. As a result, latentGLAT accomplishes a competitive performance both in decoding efficiency and quality. We propose latentGLAT, which can be directly trained without the help of knowledge distillation. Specifically, we employ discrete latent variables to capture the word categorical information and divide the original goal into the latent variables modeling and word prediction tasks. Then, we learn each task with the glancing training and encourage the model to build dependencies on the latent variables, which have fewer modes than the words and are also informative for modeling the target sentences. Experiments results on machine translation, paraphrase generation, and dialogue generation tasks validate the effectiveness of our latentGLAT. We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. This work is supported by National Science Foundation of China (No. U1836221, 6217020152).
abstain abstain objective result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective abstain other other other abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain other abstain abstain abstain abstain method abstain method method method method method abstain method abstain method method method abstain method method abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain method method method method method method abstain method method method method abstain abstain abstain abstain other abstain abstain method method method abstain method abstain abstain abstain method abstain abstain abstain method abstain method method method method method result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain method other abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other other other other other other other abstain other other other objective other objective objective method result other other other