The goal of the cross-lingual summarization (CLS) is to convert a document in one language ( e.g. , English) to a summary in another one ( e.g. , Chinese). Essentially, the CLS task is the combination of machine translation (MT) and monolingual summarization (MS), and thus there exists the hierarchical relationship between MT&MS and CLS. Existing studies on CLS mainly focus on utilizing pipeline methods or jointly training an end-to-end model through an auxiliary MT or MS objective. However, it is very challenging for the model to directly conduct CLS as it requires both the abilities to translate and summarize. To address this issue, we propose a hierarchical model for the CLS task, based on the conditional variational auto-encoder. The hierarchical model contains two kinds of latent variables at the local and global levels, respectively. At the local level, there are two latent variables, one for translation and the other for summarization. As for the global level, there is another latent variable for cross-lingual summarization conditioned on the two local-level variables. Experiments on two language directions (English  Chinese) verify the effectiveness and superiority of the proposed approach. In addition, we show that our model is able to generate better cross-lingual summaries than comparison models in the few-shot setting. The cross-lingual summarization (CLS) aims to summarize a document in source language ( e.g. , English) into a different language ( e.g. , Chinese), which can be seen as a combination of machine translation (MT) and monolingual summarization (MS) to some extent (Orasan and Chiorean, 2008; Zhu et al., 2019). The CLS can help people effectively master the core points of an article in a  Work was done when Liang and Zhou were interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China. foreign language. Under the background of globalization, it becomes more important and is now coming into widespread use in real life. Many researches have been devoted to dealing with this task. To our knowledge, they mainly fall into two categories, i.e. , pipeline and end-to-end learning methods. ( i ) The first category is pipeline-based, adopting either translation-summarization (Leuski et al., 2003; Ouyang et al., 2019) or summarization-translation (Wan et al., 2010; Orasan and Chiorean, 2008) paradigm. Although being intuitive and straightforward, they generally suffer from error propagation problem. ( ii ) The second category aims to train an end-to-end model for CLS (Zhu et al., 2019, 2020). For instance, Zhu et al. (2020) focus on using a pre-constructed probabilistic bilingual lexicon to improve the CLS model. Furthermore, some researches resort to multi-task learning (Takase and Okazaki, 2020; Bai et al., 2021a; Zhu et al., 2019; Cao et al., 2020a,b). Zhu et al. (2019) separately introduce MT and MS to improve CLS. Cao et al. (2020a,b) design several additional training objectives ( e.g. , MS, back-translation, and reconstruction) to enhance the CLS model. And Xu et al. (2020) utilize a mixed-lingual pre-training method with several auxiliary tasks for CLS. As pointed out by Cao et al. (2020a), it is challenging for the model to directly conduct CLS as it requires both the abilities to translate and summarize. Although some methods have used the related tasks ( e.g. , MT and MS) to help the CLS, the hierarchical relationship between MT&MS and CLS are not well modeled, which can explicitly enhance the CLS task. Apparently, how to effectively model the hierarchical relationship to exploit MT and MS is one of the core issues, especially when the CLS data are limited. 1 In many other related NLP tasks (Park et al., 2018; Serban et al., 2017; 1 Generally, it is difficult to acquire the CLS dataset (Zhu et al., 2020; Ayana et al., 2018; Duan et al., 2019). Shen et al., 2019, 2021), the Conditional Variational Auto-Encoder (CVAE) (Sohn et al., 2015) has shown its superiority in learning hierarchical structure with hierarchical latent variables, which is often leveraged to capture the semantic connection between the utterance and the corresponding context of conversations. Inspired by these work, we attempt to adapt CVAE to model the hierarchical relationship between MT&MS and CLS. Therefore, we propose a Variational Hierarchical Model to exploit translation and summarization simultaneously, named VHM, for CLS task in an end-to-end framework. VHM employs hierarchical latent variables based on CVAE to learn the hierarchical relationship between MT&MS and CLS. Specifically, the VHM contains two kinds of latent variables at the local and global levels, respectively. Firstly, we introduce two local variables for translation and summarization, respectively. The two local variables are constrained to reconstruct the translation and source-language summary. Then, we use the global variable to explicitly exploit the two local variables for better CLS, which is constrained to reconstruct the target-language summary. This makes sure the global variable captures its relationship with the two local variables without any loss, preventing error propagation. For inference, we use the local and global variables to assist the cross-lingual summarization process. We validate our proposed training framework on the datasets of different language pairs (Zhu et al., 2019): Zh2EnSum (Chinese  English) and En2ZhSum (English  Chinese). Experiments show that our model achieves consistent improvements on two language directions in terms of both automatic metrics and human evaluation, demonstrating its effectiveness and generalizability. Few-shot evaluation further suggests that the local and global variables enable our model to generate a satisfactory cross-lingual summaries compared to existing related methods. Our main contributions are as follows 2 :  We are the first that builds a variational hierarchical model via conditional variational auto-encoders that introduce a global variable to combine the local ones for translation and summarization at the same time for CLS. Our model gains consistent and significant performance and remarkably outperforms the 2 The code is publicly available at: https://github. com/XL2248/VHM most previous state-of-the-art methods after using mBART (Liu et al., 2020). Under the few-shot setting, our model still achieves better performance than existing approaches. Particularly, the fewer the data are, the greater the improvement we gain. Machine Translation (MT). Given an input sequence in the source language X mt = { x i } | X mt | i =1 , the goal of the neural MT model is to produce its translation in the target language Y mt = { y i } | Y mt | i =1 . The conditional distribution of the model is: p  ( Y mt | X mt ) = | Y mt | (cid:89) t =1 p  ( y t | X mt , y 1: t  1 ) , where  are model parameters and y 1: t  1 is the partial translation. Monolingual Summarization (MS). Given an input article in the source language X srcms = { x srci } | X srcms | i =1 and the corresponding summarization in the same language X tgtms = { x tgti } | X tgtms | i =1 , the monolingual summarization is formalized as: p  ( X tgtms | X srcms ) = | X tgtms | (cid:89) t =1 p  ( x tgtt | X srcms , x tgt 1: t  1 ) . CLS, we aim to learn a model that can generate a summary in the target language Y cls = { y i } | Y cls | i =1 for a given article in the source language X cls = { x i } | X cls | i =1 . Formally, it is as follows: p  ( Y cls | X cls ) = | Y cls | (cid:89) t =1 p  ( y t | X cls , y 1: t  1 ) . Conditional Variational Auto-Encoder (CVAE). The CVAE (Sohn et al., 2015) consists of one prior network and one recognition (posterior) network, where the latter takes charge of guiding the learning of prior network via KullbackLeibler (KL) divergence (Kingma and Welling, 2013). For example, the variational neural MT model (Zhang et al., 2016a; Su et al., 2018a; McCarthy et al., 2020; Su et al., 2018c), which introduces a random latent variable z into the neural MT conditional distribution: p  ( Y mt | X mt ) = (cid:90) z p  ( Y mt | X mt , z )  p  ( z | X mt ) d z . firstly sampled by the prior network from the encoder, and then the target sentence is generated by the decoder: Y mt  p  ( Y mt | X mt , z ) , where z  p  ( z | X mt ) . As it is hard to marginalize Eq. 1, the CVAE training objective is a variational lower bound of the conditional log-likelihood: L ( ,  ; X mt , Y mt ) =  KL( q  ( z  | X mt , Y mt )  p  ( z | X mt )) + E q  ( z  | X mt ,Y mt ) [log p  ( Y mt | z , X mt )]  log p ( Y mt | X mt ) , where  are parameters of the CVAE. Fig. 1 demonstrates an overview of our model, consisting of four components: encoder , variational hierarchical modules , decoder , training and inference . Specifically, we aim to explicitly exploit the MT and MS for CLS simultaneously. Therefore, we firstly use the encoder ( 3.1) to prepare the representation for the variational hierarchical module ( 3.2), which aims to learn the two local variables for the global variable in CLS. Then, we introduce the global variable into the decoder ( 3.3). Finally, we elaborate the process of our training and inference ( 3.4). Our model is based on transformer (Vaswani et al., 2017) framework. As shown in Fig. 1, the encoder takes six types of inputs, { X mt , X srcms , X cls , Y mt , X tgtms , Y cls }, among which Y mt , X tgtms , and Y cls are only for training recognition networks. Taking X mt for example, the encoder maps the input X mt into a sequence of continuous representations whose size varies with respect to the source sequence length. Specifically, the encoder consists of N e stacked layers and each layer includes two sub-layers: 3 a multi-head self-attention ( SelfAtt ) sub-layer and a position-wise feed-forward network ( FFN ) sublayer: s e = SelfAtt( h   1 e ) + h   1 e , h e = FFN( s e ) + s e , where h e denotes the state of the  -th encoder layer and h 0 e denotes the initialized embedding. Through the encoder, we prepare the representations of { X mt , X src ms , X cls } for training prior networks, encoder and decoder. Taking X mt for example, we follow Zhang et al. (2016a) and apply 3 The layer normalization is omitted for simplicity and you may refer to (Vaswani et al., 2017) for more details. mean-pooling over the output h N e ,X mt e of the N e -th encoder layer: h X mt = 1 | X mt | | X mt | (cid:88) i =1 ( h N e ,X mt e,i ) . Similarly, we obtain h X srcms and h X cls . Firstly, we design two local latent variational modules to learn the translation distribution in MT pairs and summarization distribution in MS pairs, respectively. Then, conditioned on them, we introduce a global latent variational module to explicitly exploit them. Translation. To capture the translation of the paired sentences, we introduce a local variable z mt that is responsible for generating the target information. Inspired by Wang and Wan (2019), we use isotropic Gaussian distribution as the prior distribution of z mt : p  ( z mt | X mt )  N (  mt ,  2 mt I ) , where I denotes the identity matrix and we have  mt = MLP mt ( h X mt ) ,  mt = Softplus(MLP mt ( h X mt )) , (2) where MLP(  ) and Softplus(  ) are multi-layer per-ceptron and approximation of ReLU function, respectively. At training, the posterior distribution conditions on both source input and the target reference, which provides translation information. Therefore, the prior network can learn a tailored translation distribution by approaching the recognition network via KL divergence (Kingma and Welling, 2013): q  ( z  mt | X mt , Y mt )  N (   mt ,   2 mt I ) , where   mt and   mt are calculated as:   mt = MLP mt ( h X mt ; h Y mt ) ,   mt = Softplus(MLP mt ( h X mt ; h Y mt )) , (3) where (  ;  ) indicates concatenation operation. Summarization. To capture the summarization in MS pairs, we introduce another local variable z ms , which takes charge of generating the source-language summary. Similar to z mt , we define its prior distribution as: p  ( z ms | X srcms )  N (  ms ,  2 ms I ) , where  ms and  ms are calculated as:  ms = MLP ms ( h X srcms ) ,  ms = Softplus(MLP ms ( h X srcms )) . At training, the posterior distribution conditions on both the source input and the source-language summary that contains the summarization clue, and thus is responsible for guiding the learning of the prior distribution. Specifically, we define the posterior distribution as: q  ( z  ms | X srcms , X tgtms )  N (   ms ,   2 ms I ) , where   ms and   ms are calculated as:   ms = MLP ms ( h X srcms ; h X tgtms ) ,   ms = Softplus(MLP ms ( h X srcms ; h X tgtms )) . After obtaining z mt and z ms , we introduce the global variable z cls that aims to generate a target-language summary, where the z cls can simultaneously simultaneously exploit the local variables for CLS. Specifically, we firstly encode the source input X cls and condition on both two local variables z mt and z ms , and then sample z cls . We define its prior distribution as: p  ( z cls | X cls , z mt , z ms )  N (  cls ,  2 cls I ) , where  cls and  cls are calculated as:  cls = MLP cls ( h X cls ; z mt ; z ms ) ,  cls = Softplus(MLP cls ( h X cls ; z mt ; z ms )) . At training, the posterior distribution conditions on the local variables, the CLS input, and the cross-lingual summary that contains combination information of translation and summarization. Therefore, the posterior distribution can teach the prior distribution. Specifically, we define the posterior distribution as: q  ( z  cls | X cls , z mt , z ms , Y cls )  N (   cls ,   2 cls I ) , where   cls and   cls are calculated as:   cls = MLP cls ( h X cls ; z mt ; z ms ; h Y cls ) ,   cls = Softplus(MLP cls ( h X cls ; z mt ; z ms ; h Y cls )) . The decoder adopts a similar structure to the encoder, and each of N d decoder layers includes an additional cross-attention sub-layer ( CrossAtt ): s d = SelfAtt( h   1 d ) + h   1 d , c d = CrossAtt( s d , h N e e ) + s d , h d = FFN( c d ) + c d , d As shown in Fig. 1, we firstly obtain the local two variables either from the posterior distribution predicted by recognition networks (training process as the solid grey lines) or from prior distribution predicted by prior networks (inference process as the dashed red lines). Then, conditioned on the local two variables, we generate the global variable ( z  cls / z cls ) via posterior (training) or prior (infer-ence) network. Finally, we incorporate z (  ) cls 4 into the state of the top layer of the decoder with a projection layer: o t = Tanh( W p [ h N d d,t ; z (  ) cls ] + b p ) , (8) where W p and b p are training parameters, h N d d,t is the hidden state at time-step t of the N d -th decoder layer. Then, o t is fed into a linear transformation and softmax layer to predict the probability distri-4 Here, we use z  cls when training and z cls during inference, as similar to Eq. 8. The model is trained to maximize the conditional log-likelihood, due to the intractable marginal likelihood, which is converted to the following vari-tional lower bound that needs to be maximized in the training process: J ( , ; X cls ,X mt ,X srcms ,Y cls ,Y mt ,X tgtms ) =  KL( q  ( z  mt | X mt ,Y mt )  p  ( z mt | X mt ))  KL( q  ( z  ms | X srcms ,X tgtms )  p  ( z ms | X srcms ))  KL( q  ( z  cls | X cls , z mt , z ms ,Y cls )  p  ( z cls | X cls , z mt , z ms )) + E q  [log p  ( Y mt | X mt , z mt )] + E q  [log p  ( X tgtms | X srcms , z ms )] + E q  [log p  ( Y cls | X cls , z cls , z mt , z ms )] , where the variational lower bound includes the reconstruction terms and KL divergence terms based on three hierarchical variables. We use the repa-rameterization trick (Kingma and Welling, 2013) to estimate the gradients of the prior and recognition networks (Zhao et al., 2017). During inference, firstly, the prior networks of MT and MS generate the local variables. Then, conditioned on them, the global variable is produced by prior network of CLS. Finally, only the global variable is fed into the decoder, which corresponds to red dashed arrows in Fig. 1. 4 Experiments 4.1 Datasets and Metrics Datasets. We evaluate the proposed approach on Zh2EnSum and En2ZhSum datasets released by (Zhu et al., 2019). 5 The Zh2EnSum and En2ZhSum are originally from (Hu et al., 2015) and (Hermann et al., 2015; Zhu et al., 2018), respectively. Both the Chinese-to-English and English-to-Chinese test sets are manually corrected. The involved training data in our experiments are listed in Tab. 1. Zh2EnSum. It is a Chinese-to-English summarization dataset, which has 1,699,713 Chinese short texts (104 Chinese characters on average) paired with Chinese (18 Chinese characters on average) and English short summaries (14 tokens on aver-age). The dataset is split into 1,693,713 training pairs, 3,000 validation pairs, and 3,000 test pairs. 2. En2ZhSum. It is an English-to-Chinese summarization dataset, which has 370,687 English documents (755 tokens on average) paired with multi-sentence English (55 tokens on average) and Chinese summaries (96 Chinese characters on aver-age). The dataset is split into 364,687 training pairs, 3,000 validation pairs, and 3,000 test pairs. The involved training data used in multi-task learning, model size, training time, are listed in Tab. 3. Metrics. Following Zhu et al. (2020), 1) we evaluate all models with the standard ROUGE metric (Lin, 2004), reporting the F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L. All ROUGE scores 2092 M# Models Zh2EnSum En2ZhSum RG1 RG2 RGL MVS RG1 RG2 RGL M1 GETran (Zhu et al., 2019) 24.34 9.14 20.13 0.64 28.19 11.40 25.77 M2 GLTran (Zhu et al., 2019) 35.45 16.86 31.28 16.90 32.17 13.85 29.43 M3 TNCLS (Zhu et al., 2019) 38.85 21.93 35.05 19.43 36.82 18.72 33.20 M4 ATS-A (Zhu et al., 2020) 40.68 24.12 36.97 22.15 40.47 22.21 36.89 M5 MS-CLS (Zhu et al., 2019) 40.34 22.65 36.39 21.09 38.25 20.20 34.76 M6 MT-CLS (Zhu et al., 2019) 40.25 22.58 36.21 21.06 40.23 22.32 36.59 M7 MS-CLS-Rec (Cao et al., 2020a) 40.97 23.20 36.96 NA 38.12 16.76 33.86 M8 MS-CLS* 40.44 22.19 36.32 21.01 38.26 20.07 34.49 M9 MT-CLS* 40.05 21.72 35.74 20.96 40.14 22.36 36.45 M10 MT-MS-CLS (Ours) 40.65 24.02 36.69 22.17 40.34 22.35 36.44 M11 VHM (Ours) 41.36  24.64  37.15  22.55  40.98  23.07  37.12  M12 mBART (Liu et al., 2020) 43.61 25.14 38.79 23.47 41.55 23.27 37.22 M13 MLPT (Xu et al., 2020) 43.50 25.41 29.66 NA 41.62 23.35 37.26 M14 VHM + mBART (Ours) 43.97  25.61  39.19  23.88 41.95  23.54  37.67  Table 4: ROUGE F1 scores (%) and MoverScore scores (%) on Zh2EnSum test set, and ROUGE F1 scores (%) on En2ZhSum test set. are reported by the 95% confidence interval measured by the official script; 6 2) we also evaluate the quality of English summaries in Zh2EnSum with MoverScore (Zhao et al., 2019). In this paper, we train all models using standard transformer (Vaswani et al., 2017) in Base setting. For other hyper-parameters, we mainly follow the setting described in Zhu et al. (2019, 2020) for fair comparison. For more details, please refer to Appendix A. 4.3 Comparison Models Pipeline Models. TETran (Zhu et al., 2019). It first translates the original article into the target language by Google Translator 7 and then summarizes the translated text via LexRank (Erkan and Radev, 2004). TLTran (Zhu et al., 2019). It first summarizes the original article via a transformer-based monolingual summarization model and then translates the summary into the target language by Google Translator. End-to-End Models. TNCLS (Zhu et al., 2019). It directly uses the de-facto transformer (Vaswani 6 The parameter for ROUGE script here is -c 95 -r 1000 -n 2 -a 7 https://translate.google.com/ et al., 2017) to train an end-to-end CLS system. ATS-A (Zhu et al., 2020). 8 It is an efficient model to attend the pre-constructed probabilistic bilingual lexicon to enhance the CLS. MS-CLS (Zhu et al., 2019). It simultaneously performs summarization generation for both CLS and MS tasks and calculates the total losses. MT-CLS (Zhu et al., 2019). 9 It alternatively trains CLS and MT tasks. MS-CLS-Rec (Cao et al., 2020a). It jointly trains MS and CLS systems with a reconstruction loss to mutually map the source and target representations. mBART (Liu et al., 2020). We use mBART ( mbart.cc 25 ) as model initialization to fine-tune the CLS task. MLPT (Mixed-Lingual Pretraining) (Xu et al., 2020). It applies mixed-lingual pretraining that leverages six related tasks, covering both cross-lingual tasks such as translation and monolingual tasks like masked language models. MT-MS-CLS. It is our strong baseline, which is implemented by alternatively training CLS, MT, and MS. Here, we keep the dataset used for MT and MS consistent with Zhu et al. (2019) for fair comparison. Overall, we separate the models into three parts in Tab. 4: the pipeline, end-to-end, and multi-task 8 https://github.com/ZNLP/ATSum 9 https://github.com/ZNLP/NCLS-Corpora 2093      3URSRUWLRQRI8VHG&/67UDLQLQJ'DWD          5 *  =K(Q6XP 9+00706&/607&/606&/6$76$*DS+*DS'      3URSRUWLRQRI8VHG&/67UDLQLQJ'DWD       5 *  =K(Q6XP 9+00706&/607&/606&/6$76$*DS+*DS'      3URSRUWLRQRI8VHG&/67UDLQLQJ'DWD         5 * / =K(Q6XP 9+00706&/607&/606&/6$76$*DS+*DS'      3URSRUWLRQRI8VHG&/67UDLQLQJ'DWD      0 9 6 =K(Q6XP 9+00706&/607&/606&/6$76$*DS+*DS' Figure 2: ROUGE F1 scores (%) and MoverScore scores (%) on Zh2EnSum test set in few-shot setting. settings. In each part, we show the results of existing studies and our re-implemented baselines and our approach, i.e. , the VHM, on Zh2EnSum and En2ZhSum test sets. Results on Zh2EnSum. Compared against the pipeline and end-to-end methods, VHM substantially outperforms all of them ( e.g. , the previous best model ATS-A) by a large margin with 0.68/0.52/0.18/0.4  scores on RG1/RG2/RGL/MVS, respectively. Under the multi-task setting, compared to the existing best model MS-CLS-Rec, our VHM also consistently boosts the performance in three metrics ( i.e. , 0.39  , 1.44  , and 0.19  ROUGE scores on RG1/RG2/RGL, respec-tively), showing its effectiveness. Our VHM also significantly surpasses our strong baseline MT-MS-CLS by 0.71/0.62/0.46/0.38  scores on RG1/RG2/RGL/MVS, respectively, demonstrating the superiority of our model again. Results on En2ZhSum. Compared against the pipeline, end-to-end and multi-task methods, our VHM presents remarkable ROUGE improvements over the existing best model ATS-A by a large margin, about 0.51/0.86/0.23  ROUGE gains on RG1/RG2/RGL, respectively. These results suggest that VHM consistently performs well in different language directions. Our approach still notably surpasses our strong baseline MT-MS-CLS in terms of all metrics, which shows the generalizability and superiority of our model again. Due to the difficulty of acquiring the cross-lingual summarization dataset (Zhu et al., 2019), we conduct such experiments to investigate the model performance when the CLS training dataset is limited, i.e. , few-shot experiments. Specifically, we randomly choose 0.1%, 1%, 10%, and 50% CLS training datasets to conduct experiments. The results are shown in Fig. 2 and Fig. 3. Results on Zh2EnSum. Fig. 2 shows that VHM significantly surpasses all comparison models under each setting. Particularly, under the 0.1% setting, our model still achieves best performances than all baselines, suggesting that our variational hierarchical model works well in the few-shot setting as well. Besides, we find that the performance gap between comparison models and VHM is growing when the used CLS training data become fewer. It is because relatively larger proportion of translation and summarization data are used, the influence from MT and MS becomes greater, effectively strengthening the CLS model. Particularly, the performance Gap-H between MT-MS-CLS and VHM is also growing, where both models utilize the same data. This shows that the hierarchical relationship between MT&MS and CLS makes substantial contributions to the VHM model in terms of four metrics. Consequently, our VHM achieves a comparably stable performance. Results on En2ZhSum. From Fig. 3, we observe the similar findings on Zh2EnSum. This shows that VHM significantly outperforms all comparison models under each setting, showing the generalizability and superiority of our model again in the few-shot setting. We conduct ablation studies to investigate how well the local and global variables of our VHM works. When removing variables listed in Tab. 5, we have the following findings. (1) Rows 1  3 vs. row 0 shows that the model performs worse, especially when removing the two local ones (row 3), due to missing the explicit translation or summarization or both information provided by the local variables, which is important to CLS. Besides, row 3 indicates that directly attending to z cls leads to poor performances, showing the necessity of the hierarchical structure, i.e. , using the global variable to exploit the local ones. (2) Rows 4  5 vs. row 0 shows that directly attending the local translation and summarization cannot achieve good results due to lacking of the global combination of them, showing that it is very necessary for designing the variational hierarchical model, i.e. , using a global variable to well exploit and combine the local ones. Following Zhu et al. (2019, 2020), we conduct human evaluation on 25 random samples from each of the Zh2EnSum and En2ZhSum test set. We compare the summaries generated by our methods (MT-MS-CLS and VHM) with the summaries generated by ATS-A, MS-CLS, and MT-CLS in the full setting and few-shot setting (0.1%), respectively. We invite three graduate students to compare the generated summaries with human-corrected references, and assess each summary from three independent perspectives: 1. How informative ( i.e. , IF) the summary is? 2. How concise ( i.e. , CC) the summary is? 3. How fluent , grammatical ( i.e. , FL) the summary is? Each property is assessed with a score from 1 (worst) to 5 (best). The average results are presented in Tab. 6 and Tab. 7. Tab. 6 shows the results in the full setting. We find that our VHM outperforms all comparison models from three aspects in both language directions, which further demonstrates the effectiveness and superiority of our model. Tab. 7 shows the results in the few-shot setting, where only 0.1% CLS training data are used in all models. We find that our VHM still performs best than all other models from three perspectives in both datasets, suggesting its generalizability and effectiveness again under different settings. Cross-Lingual Summarization. Conventional cross-lingual summarization methods mainly focus on incorporating bilingual information into 2095 Models Zh2EnSum En2ZhSum IF CC FL IF CC FL ATS-A 3.44 4.16 3.98 3.12 3.31 3.28 MS-CLS 3.12 4.08 3.76 3.04 3.22 3.12 MT-CLS 3.36 4.24 4.14 3.18 3.46 3.36 MT-MS-CLS 3.42 4.46 4.22 3.24 3.48 3.42 VHM 3.56 4.54 4.38 3.36 3.54 3.48 Table 6: Human evaluation results in the full setting. the pipeline methods (Leuski et al., 2003; Ouyang et al., 2019; Or  asan and Chiorean, 2008; Wan et al., 2010; Wan, 2011; Yao et al., 2015; Zhang et al., 2016b), i.e. , translation and then summarization or summarization and then translation. Due to the difficulty of acquiring cross-lingual summarization dataset, some previous researches focus on constructing datasets (Ladhak et al., 2020; Scialom et al., 2020; Yela-Bello et al., 2021; Zhu et al., 2019; Hasan et al., 2021; Perez-Beltrachini and Lapata, 2021; Varab and Schluter, 2021), mixed-lingual pre-training (Xu et al., 2020), knowledge distillation (Nguyen and Tuan, 2021), contrastive learning (Wang et al., 2021) or zero-shot approaches (Ayana et al., 2018; Duan et al., 2019; Dou et al., 2020), i.e. , using machine translation (MT) or monolingual summarization (MS) or both to train the CLS system. Among them, Zhu et al. (2019) propose to use roundtrip translation strategy to obtain large-scale CLS datasets and then present two multi-task learning methods for CLS. Based on this dataset, Zhu et al. (2020) leverage an end-to-end model to attend the pre-constructed probabilistic bilingual lexicon to improve CLS. To further enhance CLS, some studies resort to shared decoder (Bai et al., 2021a), more pseudo training data (Takase and Okazaki, 2020), or more related task training (Cao et al., 2020b,a; Bai et al., 2021b). Wang et al. (2022) concentrate on building a benchmark dataset for CLS on dialogue field. Different from them, we propose a variational hierarchical model that introduces a global variable to simultaneously exploit and combine the local translation variable in MT pairs and local summarization variable in MS pais for CLS, achieving better results. Conditional Variational Auto-Encoder. CVAE has verified its superiority in many fields (Sohn et al., 2015; Liang et al., 2021a; Zhang et al., 2016a; Su et al., 2018b). For instance, in dialogue, Shen et al. (2019), Park et al. (2018) and Serban et al. (2017) extend CVAE to capture the semantic connection between the utterance and the corresponding context with hierarchical latent variables. Although the CVAE has been widely used in NLP tasks, its adaption and utilization to cross-lingual summarization for modeling hierarchical relationship are non-trivial, and to the best of our knowledge, has never been investigated before in CLS. Multi-Task Learning. Conventional multi-task learning (MTL) (Caruana, 1997), which trains the model on multiple related tasks to promote the representation learning and generalization performance, has been successfully used in NLP fields (Collobert and Weston, 2008; Deng et al., 2013; Liang et al., 2021d,c,b). In the CLS, conventional MTL has been explored to incorporate additional training data (MS, MT) into models (Zhu et al., 2019; Takase and Okazaki, 2020; Cao et al., 2020a). In this work, we instead focus on how to connect the relation between the auxiliary tasks at training to make the most of them for better CLS. In this paper, we propose to enhance the CLS model by simultaneously exploiting MT and MS. Given the hierarchical relationship between MT&MS and CLS, we propose a variational hierarchical model to explicitly exploit and combine them in CLS process. Experiments on Zh2EnSum and En2ZhSum show that our model significantly improves the quality of cross-lingual summaries in terms of automatic metrics and human evaluations. Particularly, our model in the few-shot setting still works better, suggesting its superiority and generalizability. The research work descried in this paper has been supported by the National Key R&D Program of China (2020AAA0108001) and the National Nature Science Foundation of China (No. 61976015, 61976016, 61876198 and 61370130). Liang is supported by 2021 Tencent Rhino-Bird Research Elite Training Program. The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve this paper.
abstain abstain abstain abstain objective abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain method abstain method abstain objective method objective other abstain result abstain other other other other other abstain abstain abstain abstain other abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain other method method abstain abstain method abstain abstain method method abstain other abstain method method abstain method abstain abstain method method abstain abstain method abstain abstain abstain abstain method method method abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other method method method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain other abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain result abstain result result abstain abstain method abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other objective other other other objective other other other method objective result abstain other other other