Various models have been proposed to incorporate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks. 1 1 Introduction Syntactic structures serve as the principle of how words are correctly combined to form sentences. It is widely acknowledged that learning syntactic structures should improve neural text generation (Shen et al., 2018; Peng et al., 2019; Du et al., 2020). Even though current neural language models, such as Transformer (Vaswani et al., 2017) and GPT-2 (Radford et al., 2019) have achieved outstanding performance without explicitly modeling latent syntactic structures, these models still fail to learn the long-range syntactic dependencies (Kun-coro et al., 2018; Xu et al., 2021). To leverage explicit syntactic knowledge in natural language generation (NLG), many methods have been proposed (Wu et al., 2017; Shen et al., 2018; Zhang et al., 2019; Kim et al., 2019; Du 1 Our code is available at https: //github.com/FadedCosine/Dependency-Guided-Neural-Text-Generation et al., 2020). We conclude from previous works that knowledge of syntactic structures can bring four advantages to neural language models: (1) Syntactic structures can be modeled to obtain better representations of natural language sentences (Jacob et al., 2018; Williams et al., 2018; Wang et al., 2019). (2) Jointly training syntactic structure parsing and language modeling can contribute to each other (Shen et al., 2018; Dyer et al., 2016; Kim et al., 2019; Du et al., 2020; Shen et al., 2021b). (3) Syntactic structures can be used to directly model the composition of language (Socher et al., 2013; Casas et al., 2020) and help with the long-range dependency problem by providing shortcuts for gradient backpropagation (Chung et al., 2017). (4) Integrating syntactic structures into a neural network can improve generalization via a better inductive bias (Shen et al., 2019; Zhang et al., 2019). Despite these advantages, it is not trivial to incorporate knowledge of syntactic structures into neural language models effectively and efficiently. Several practical problems arise: (1) Previous works (Chung et al., 2017; Shen et al., 2018; Dyer et al., 2016; Kim et al., 2019; Shen et al., 2019) have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN) (Sutskever et al., 2014). These methods are difficult to be adapted to other neural language models, such as Transformer and GPT-2. (2) If jointly modeling language modeling and syntactic structure parsing, it will require much more time/memory during training or inference. To address these problems while keeping the advantages, we explore incorporating knowledge of syntactic structures in a different manner. In this work, we propose a novel dependency modeling objective to train neural language models to directly predict the current token's future dependent tokens given the history. We define the future dependent to-7758 Models External Parameters? kens of a specific token in a sentence as its children and parent in the dependency parse tree that will appear in the rest of the sentence. Further, we propose Dependency-based Mixture Language Models (DMLM) that, at each timestep, mixes the previous dependency modeling probability distributions with self-attention to get the next-token probability. As shown in Table 1, the proposed method can be adapted to any neural language model without adding external networks or parameters. Our core idea can be illustrated in Figure 1 and Figure 2: when predicting the next-token "indi-cate" after reading "red figures on the screen", common language models are easy to predict an incorrect word, such as "indicates", since the prediction of these models relies heavily on the recent word, "screen" in this case. However, our propose DMLM will directly look back into the long-range context, and select the next-token from all the future dependent tokens predicted by previous tokens. According to the underlying dependency structure, DMLM pays different weights to different tokens' future dependent tokens. Thus, the model is more likely to predict "indicate" since DMLM tends to think of the next-token as a future dependent token of "figures" rather than "screen". We conduct experiments with different neural language models including LSTM (Hochreiter and Schmidhuber, 1997), Transformer (Vaswani et al., 2017), and GPT-2 (Radford et al., 2019) across different tasks in conditional text generation, unconditional text generation, and language modeling. Through extensive experiments we demonstrate that DMLM consistently improves the generation quality according to both human evaluations and automatic metrics. Compared to other neural language models that incorporate syntactic knowledge, indicate figures stocks red screen falling on the ROOTROOT red figures on the screen indicate falling stocks . DMLM is architecturally simpler and easier to fit into any neural language model, while possessing wide applicability to different text generation tasks. Our goal is to propose a simple yet effective method that can improve neural text generation by learning from the underlying syntactic structure, and can fit into any auto-regressive generation model without using additional elaborate components. We first introduce a novel dependency modeling objective to force the model to directly predict the future dependent tokens of the current token. Based on the dependency modeling, we then present the proposed DMLM. It has been a challenge to equip neural language models with the capability of modeling long-range dependency in text (Dai et al., 2019). In particular, previous works (Wu et al., 2017) observe that vanilla RNN can hardly capture many subtle long-range token dependencies effectively. On 7759 red <BOS> red figures figures on on the the screen screen + A tt e n ti on D i s t r i bu ti on D e p e nd e n c y M od e li ng D i s t r i bu ti on s       indicate F i n a l P r e d i c ti on Figure 2: Illustration of DMLM. the other hand, though self-attention mechanisms can build direct connections between long-distance token pairs, it is still elusive for Transformer to be aware of syntactic dependency structures while also obtaining strong language modeling performance (Shen et al., 2021a). The current neural language models are mostly trained purely using the language modeling objective with Maximum Likelihood Estimation (MLE). With the auto-regressive factorization, language modeling can be reduced to modeling the conditional distribution of the next-token x t given the context x <t = { x 1 , . . . , x t  2 , x t  1 } . However, in order to make neural language models aware of long-range dependency and syntactic structures, we propose the dependency modeling objective to train models to learn the probability distribution of the future dependent tokens directly. Following Ahmed et al. (2019), we define the future dependent tokens of a specific token in a sentence as its children and parent in the dependency parse tree that will appear in the rest of the sentence. Taking Figure 1 as an example, the future dependent tokens of "figures" are "screen" and "indicate", since "red" does not appear after "figures" in this sentence. Specifically, given a token sequence x = { x 1 , . . . , x T  1 , x T } where T  N denotes the sequence length, we first use dependency parser to generate a dependency tree. Then, we derive the future dependent tokens set Z t for each token x t  1 , where Z t = { x i | i  t, x i is the child or parent of x t  1 } . We train a language model  to maximize the log-likelihood sum of tokens in Z t . This equals to minimize: LDM (  ) =  T (cid:88) t =1 (cid:88) z t  Z t log p dep  ( z t | x <t ) , (1) which is the dependency modeling objective. To give a categorical probability distribution over the next-token, a standard approach for the current neural language models is to encode the context into a fixed-size vector followed by an output embedding layer and a softmax function. In our case, given the context x <t , we first train the language model to directly learn the probability distribution of x t  1 's future dependent tokens p dep  ( w | x <t ) by dependency modeling (Section 2.1). We then propose DMLM (depicted in Figure 2) that mixes dependency modeling probability distributions P dep = { p dep  ( w | x < 1 ) , . . . , p dep  ( w | x <t  1 ) , p dep  ( w | x <t ) } . All the probability distributions in P dep are weighed by self-attention, and summed to obtain the final next-token probability distribution. We can easily implement a self-attention in both Transformer-based and RNN-based language models. For example, in Transformer and GPT-2, the penultimate layer seems to naturally learn alignments (Garg et al., 2019), so we use its average attention weights over all the attentions heads as the dependency attention distribution. In RNN-based models, inspired by Merity et al. (2017) and 7760 Vaswani et al. (2017), at each timestep, we linearly project the current hidden state h t  RH to a query vector q t = WQ h t and a key vector k t = WK h t , where WQ  RH  H , WK  RH  H , q t  RH , and k t  RH . To generate the dependency attention, we compute the match between the query q t and the context's keys { k 1 , . . . , k t  1 , k t } by taking the inner product, followed by a softmax to obtain the dependency attention distribution: e (t) = { e ( t ) 1 , . . . , e ( t ) t  1 , e ( t ) t } , e ( t ) i = q Tt k i , 1  i  t, a ( t ) = softmax ( e (t)  H ) , a ( t ) = { a ( t ) 1 , . . . , a ( t ) t  1 , a ( t ) t } , (2) where e (t)  R t , and a ( t )  R t . H The dependency attention distribution reveals which token in the context may have a strong dependency relation with the token to be predicted. Thus, the neural language model should pay more attention to previous tokens with high dependency attention scores, i.e., the next-token is more likely to be the future dependent token of those tokens in the context. Formally, the next-token probability is the sum of the context's dependency modeling probability distributions weighed by the dependency attention scores: p  ( w | x <t ) = t (cid:88)  =1 a ( t )  p dep  ( w | x < ) . where p dep  ( w | x < ) is the probability distribution of x   1 's future dependent tokens, since till now the neural language model is only trained by dependency modeling. Then, we further finetune the neural language model using MLE, but with respect to our modified probability distribution given in Equation 3: LLM (  ) =  T (cid:88) t =1 log p  ( x t | x <t ) . For each timestep during inference, DMLM outputs a dependency modeling distribution, and we store it in a list. To predict the next-token, DMLM applies self-attention in Equation 2 to produce a dependency attention distribution over the context, and then the next-token probability can be calculated by Equation 3, where the list preserves all the p dep  ( w | x < ) , 1    t . Despite previous works mainly focusing on language modeling, it has always been a thorny issue whether better language models lead to better performance in downstream tasks. Therefore, we showcase the performance of our proposed DMLM in three different tasks: conditional text generation (Section 3.1), unconditional text generation (Sec-tion 3.2), and language modeling (Section 3.3). To verify the effectiveness and architecturally generalizability of our method, we conduct the generation tasks with three dominant neural language models, including LSTM, Transformer and GPT-2. We prefix the base model name with " DM" to denote the corresponding Dependency-based Mixture language model. Specifically, we adopt AWD-LSTM (Merity et al., 2018) as our base LSTM, and further compare our DM-LSTM with PRPN (Shen et al., 2018) and ON-LSTM (Shen et al., 2019) which also incorporate knowledge of syntactic structures, and are built on LSTM. In the same task, we use exactly the same hyper-parameters and setups for the pairs of base models and corresponding DM-models. Other details of the experimental setup for each task can be seen in Appendix A. For all the tasks, we use a state-of-the-art parser, HPSG Parser 2 (Zhou and Zhao, 2019) to get the dependency parse tree for each sentence in the datasets. We discuss the impact of the dependency parser in Appendix B. 3.1 Conditional Text Generation Setup We take the story ending generation as the conditional text generation task, and evaluate our method on the ROCStories corpus (Mostafazadeh et al., 2016), which consists of 98,161 five-sentences. We follow the preprocessing 3 of Kong et al. (2021) to randomly split ROCStories by 8:1:1 for training/validation/test, respectively, and delexicalize stories by masking all the male/female/unknown names with "[MALE]"/"[FEMALE]"/"[NEUTRAL]". We finally get a word-level vocabulary with 31 , 216 unique tokens. The conditional text generation task is to generate a reasonable ending given a four-sentence story context. For all models, we generate stories using nucleus sampling (Holtzman et al., 2 https://github.com/DoodleJZ/ HPSG-Neural-Parser 3 We use the preprocessed data in https://github.com/thu-coai/Stylized-Story-Generation-with-Style-Guided-Planning 7761 Models UNION  BERTScore  B-1  B-2  D2  D3  SB-2  SB-3  PRPN 83.37 29.11 21.45 6.84 13.22 33.50 95.17 86.76 ON-LSTM 82.18 29.41 22.16 7.33 13.93 35.71 94.98 85.80 AWD-LSTM 82.98 29.57 22.23 7.31 14.07 35.71 94.92 85.88 DM-LSTM 83.97  29.93 22.54  7.63  14.92 37.44 94.47  84.77  Transformer 81.39 27.64 21.28 7.01 17.48 42.30 93.18 81.52 DM-Transformer 84.07  28.20  21.49 7.29  17.79 42.08 92.86  81.36  GPT-2 84.41 29.02 21.79 7.45 17.09 40.74 93.51 82.55 DM-GPT-2 85.31  30.18  22.81  8.02  17.98 43.29 93.18 81.41  Table 2: Automatic evaluation results for the conditional text generation task on Rocstories dataset. 2020) with p = 0 . 5 . We measure the generated story endings by the following automatics metrics: (1) UNION (Guan and Huang, 2020): It is a learnable unreferenced metric for evaluating the quality of generated stories; (2) BERTScore (Zhang et al., 2020): The metric measures the semantic consistency between the generated and the referenced ones by BERT (De-vlin et al., 2019); (3) BLEU (B-n) (Papineni et al., 2002): BLEU evaluates n -gram overlap between the generated stories and the references; (4) Distinct (D-n) (Li et al., 2016): The proportions of distinct n -grams in the outputs to evaluate the diversity of generated results. Since Distinct score will become extremely low for small n , we calculate it with n = 2 , 3 ; (5) Self-BLEU (SB-n) (Zhu et al., 2018): The metric is calculated by computing n grams ( n = 2 , 3 ) BLEU score of each generated text with all other generated ones as references. Smaller Self-BLEU scores indicate better diversity. Results The experimental results of baselines and corresponding DM-models are shown in Table 2. Note that we do not conduct significant tests on Distinct since it is a document-level metric. We can see that, all the DM-models significantly outperform baseline models on almost all the metrics. Furthermore, compared with PRPN and ON-LSTM, our DM-LSTM performs signifi-Models LM score  RLM score  PRPN 5.24 5.75 ON-LSTM 5.20 5.59 AWD-LSTM 5.18 5.64 DM-LSTM 5.14 5.52 Transformer 5.00 5.59 DM-Transformer 4.97 5.49 GPT-2 4.89 5.55 DM-GPT-2 4.67 5.47 Table 4: Results of global metrics for the unconditional text generation task on EMNLP2017 WMT News. cantly better in all the metrics. This indicates that incorporating knowledge of syntactic structures in our proposed way can effectively contribute to both the quality and diversity of the story ending generation. Moreover, no matter what the base model is, our DM-model can substantially improves the conditional text generation. This demonstrates that our method can be effectively adapted to different neural language models, such as the large scale language model, GPT-2, while previous models like ON-LSTM can only be built on LSTM. Human evaluation To further evaluate the fluency and logic of generated stories, following (Guan et al., 2020), we conduct pair-wise comparisons between DM-models and corresponding 7762 Models Nucleusp 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 PRPN 41.48 45.77 55.32 64.23 83.98 109.3 172.09 302.57 ON-LSTM 37.46 42.98 46.16 56.69 72.36 98.06 152.60 274.43 AWD-LSTM 37.97 41.80 48.74 57.45 71.77 94.22 146.40 289.13 DM-LSTM 36.11 39.53  47.67 55.30 69.38 95.95 136.98  256.51  Transformer 45.37 46.36 50.90 60.27 70.74 91.65 125.46 222.27 DM-Transformer 37.74  40.75  43.25  49.92  60.28  76.77  104.03  182.29  GPT-2 41.19 44.05 47.86 53.97 63.18 81.45 112.81 192.10 DM-GPT-2 36.41  40.99  41.75  46.18  55.36  67.97  92.22  152.98  Table 5: GPT-2 Perplexity on 1 , 000 random samples with various sampling hyper-parameters generated by models trained on EMNLP2017 WMT News dataset. baselines. We randomly sample 100 story endings from each model. For each pair of stories (one by the DM-model and the other by the baseline, along with the beginning), five annotators are hired to give a preference (win, lose, or tie) from the following two aspects: (1) Grammaticality : whether a story ending is natural and fluent; (2) Logicality : whether a story is coherent to the given beginning and reasonable in terms of causal and temporal dependencies in the context. The detailed questionnaire and other details are shown in Appendix D. The average win/lose/tie rates of the human evaluation are shown in Table 3. To measure the inter-annotator agreement, we calculate Krippendorff's alpha (Hayes and Krippendorff, 2007) for each pair-wise comparison, and all the results are fair agreement ( 0 . 2    0 . 4 ) or moderate agreement ( 0 . 4    0 . 6 ). The results show that our DM-models significantly outperform baseline models in both the grammaticality and logicality. Setup We perform experiments of unconditional text generation on EMNLP2017 WMT News dataset 4 . We use the preprocessed data of a recent work 5 (Caccia et al., 2020) that contains 5 , 268 distinct words with maximum sentence length 51. The training/validation/test set consists of 268 , 586 / 10 , 000 / 10 , 000 sentences. Following Caccia et al. (2020), we evaluate the models with the global metrics (Semeniuta et al., 2018): (1) Language Model score (LM score) : We use the oracle Language Model to evaluate the negative log-likelihood of generated text as the metric to reflect quality; (2) Reverse Language Model score (RLM score) We train a new Language Model on the generated text, and then evaluate the negative log-likelihood of a held-out set of real text. This metric can measure text diversity since the generated text with better diversity would have a broader coverage over the real data space, and the new Language Model can be trained better, thus leading to lower RLM score. Both the LM score and RLM score are usually evaluated on the sentences generated by purely random sampling. Besides, to further measure the generation fluency, we directly use the public GPT-2 checkpoint of pretrained parameters without finetuning to calculate GPT-2 Perplexity of generated samples. Results Table 4 shows the results of global metrics obtained by various models. All the DM-models again outperform the baselines. The consistently lower LM scores indicate that the generated 4 http://statmt.org/wmt17/ translation-task.htm 5 https://github.com/pclucas14/ GansFallingShort/tree/master/real_data_experiments/data/news 7763 Models #Params Dev PPL Test PPL Pointer Sentinel-LSTM (Merity et al., 2017) 21M 72.4 70.9 RNNG (Dyer et al., 2016) -88.7 Variational RHN (Zilly et al., 2017) 23M 67.9 65.4 PRPN (Shen et al., 2018) -62.0 Fraternal dropout (Zolna et al., 2018) 24M 58.9 56.8 URNNG (Kim et al., 2019) -85.9 ON-LSTM (Shen et al., 2019) 25M 58.3 56.2 AWD-LSTM (Merity et al., 2018) 24M 60.0 57.3 DM-LSTM (Ours) 24M 58.6 56.2 AWD-LSTM-MoS(Yang et al., 2018) 22M 56.5 54.4 AWD-LSTM-DOC(Takase et al., 2018) 23M 54.1 52.4 Table 7: Various language models' perplexity evaluated on validation and test sets of Penn Treebank dataset. sentences of DM-models are of better quality, while the consistently lower RLM scores also demonstrate that DM-models can generate more diverse sentences meanwhile. In addition, each model is used to generate 1 , 000 sentences with various sampling hyper-parameters, and GPT-2 Perplexity is further calculated. As shown in Table 5, our proposed method can make neural language models perform significantly better in terms of generation fluency. In particular, Transformer-based models can gain more significant improvement from DMLM. We conjecture that this is because, in our implementation, we directly uses the penultimate multi-head attention layer of Transformer to obtain the dependency attention distribution of DMLM. Thus, it can easily inherit all the strengths of Transformer-based models. Human evaluation Following previous work (Yu et al., 2017; Guo et al., 2018), we conduct a Turing test to further evaluate the generated text. In practice, we mix 100 randomly sampled sentences from each model, and another 100 sentences from the real test set. Five annotators are hired to judge whether each of the 900 sentences is created by human or machines. Each sentence gets +1 score when it is regarded as a real one, and 0 score otherwise. The detailed questionnaire and other details are shown in Appendix D. The average score for each model is shown in Table 6, from which we can see all the DM-models surpass the baselines. Both automatic evaluations and human evaluations indicate that DMLM can help neural language models generate more readable, fluent, and natural sentences. Setup We evaluate the proposed method with the word-level language modeling task by measuring Perplexity (PPL) on the Penn Treebank (PTB) (Marcus et al., 1993; Mikolov et al., 2012) corpora. The PTB dataset has a vocabulary size of 10 , 000 unique words, and the training/validation/test set consists of 42 , 068 / 3 , 370 / 3 , 761 sentences. For this task, we mainly implement the DMLM on the RNN-based language model, i.e., AWD-LSTM (Merity et al., 2018). For a fair comparison, our DM-LSTM uses exactly the same hyper-parameters and setups as AWD-LSTM. Since Transformer-based models' strong performance relies on training with large datasets, it will perform worse than random when trained on a small dataset (Shen et al., 2021a). We still report Transformer-based models' language modeling results on PTB in Appendix C. Results We compare our method with its base model, AWD-LSTM, and we report the results along with other state-of-the-art models in Table 7. Compared with the AWD-LSTM, our DM-LSTM reduces the perplexity by 1.4 on the validation set and 1.1 on the test set, indicating that incorporating knowledge of syntactic structures in our proposed manner can substantially improve language modeling. Compared with other models that also leverage syntactic knowledge, our DM-LSTM strongly outperforms RNNG, PRPN, and URNNG. Moreover, though DM-LSTM does not make any changes to the architecture of the AWD-LSTM language model, it still achieves a comparable perplexity with ON-LSTM. Note that, since our method is model-agnostic, it can be harmonically 7764 5 2 2 7 U H G I L JX U H V RQ W K H V F U HH Q L QG L FD W H I D OOL QJ V W R F N V UHG ILJXUHV RQ WKH VFUHHQ LQGLFDWH IDOOLQJ VWRFNV  10 10 10 8 10 6 10 4 10 2 10 0 Figure 3: Visualization of dependency attention distributions. We show how our proposed method works by visualizing the dependency attention distributions. We use DM-Transformer to generate a sentence: "red figures on the screen indicate falling stocks." For each generation step, we record this step's dependency attention distribution. When we finally generate the whole sentence, we get 9 distributions and plot Figure 3 from them. Each row in Figure 3 shows the dependency attention distribution of the model when generating the corresponding Y-axis token. When predicting the token "indicate", DMLM pays great attention to "figures". This is because these two tokens have a direct dependency connection in the dependency parse tree, and our method successfully captures this relationship. In addition, DMLM also helps the model better organize dependency information when the next-tokens, such as "screen" and "stocks", have dependencies on more than one token in the context. We perform case studies for a better understanding of the model performance. Table 8 provides examples of conditional text generation produced by our DM-models and other baselines. Obviously, all the DM-models can generate more reasonable and coherent story endings. Additionally, some examples of unconditional text generation are shown in Table 9 and Appendix E. These examples show that our DMLM can help base models generate more reasonable, readable, fluent, and natural sentences. Compared with vanilla RNN, our DM-RNN indeed increases the computational complexity from O ( T ) to O ( T 2 ) . In practice, we can follow Merity et al. (2017) to set a context window that allows DMLM looks L timesteps into the past at most, where L is the context length. However, our DMLM can efficiently apply to Transformer-based models without additional computational complexity. Many previous studies have shown that leveraging the knowledge of syntactic structures can improve NLG (Chelba, 1997; Roark, 2001; Emami and Je-linek, 2005; Buys and Blunsom, 2015). Mirowski and Vlachos (2015) incorporated syntactic dependencies into the RNN formulation, but they limited the scope to the scoring of complete sentences, not to next word prediction. Some other efforts have been done to integrate dependency structure into neural machine translation (NMT) from both the source and target side. Eriguchi et al. (2016) proposed a tree-to-sequence attentional NMT model where source-side parse tree was used. Wu et al. (2017) involved target syntactic trees into NMT model to jointly learn target translation and dependency parsing. Casas et al. (2020) introduced a syntactic inductive bias to NLG in an iterative non-autoregressive way. For neural language models, recently, Dyer et al. (2016) proposed recurrent neural network grammar (RNNG) to jointly model syntax and surface structure by incrementally generating a syntax tree and sentence. Subsequent work (Kim et al., 2019) extended the model to an unsupervised version. Shen et al. (2018) introduced the Parsing-Reading-Predict Networks (PRPN) to calculate syntactic distances among words and use self-attention to compose previous states. Its subsequent work (Shen et al., 2019) transferred the distance notion to LSTM cell, and introduced Ordered Neurons LSTM (ON-LSTM). edge of syntactic structures by introducing complex architectural changes. Therefore, it can get very unwieldy to adapt them to other neural language models, such as Transformer and GPT-2. In this paper, we introduce Dependency-based Mixture Language Models, which can incorporate knowledge of dependency structures into arbitrary auto-regressive generation models without any changes to the original architectures. Both automatic and human evaluation results in extensive experiments across different tasks and different architectures demonstrate the effectiveness and generalizability of our method. In the future, we will explore to incorporate the dependency labels into our method, and combine our DMLM with more neural language models. Second, we would like to integrate other linguistic knowledge, such as constituency structures and semantic information, into neural language models in our manner. This work was supported by National Key R&D Program of China (No.2018YFB1005100), Bejing Academy of Artificial Intelligence (BAAI) and State Key Laboratory of Media Convergence Production Technology and Systems. We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
abstain abstain abstain objective method result abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain objective objective method abstain objective abstain method objective abstain abstain method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method method other other abstain abstain abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain other abstain abstain result abstain abstain abstain result result result abstain abstain result abstain other abstain abstain other abstain result result abstain abstain abstain other abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain result abstain other abstain method abstain method abstain abstain abstain abstain method method abstain result method method method other other other other other other other other other other other other abstain objective objective abstain other other other