Generating long and coherent text is an important but challenging task, particularly for open-ended language generation tasks such as story generation. Despite the success in modeling intra-sentence coherence, existing generation models (e.g., BART) still struggle to maintain a coherent event sequence throughout the generated text. We conjecture that this is because of the difficulty for the decoder to capture the high-level semantics and discourse structures in the context beyond token-level co-occurrence. In this paper, we propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. To this end, we propose two pretraining objectives to learn the representations by predicting inter-sentence semantic similarity and distinguishing between normal and shuffled sentence orders. Extensive experiments show that our model can generate more coherent texts than state-of-the-art baselines. The ability to generate coherent long texts plays an important role in many natural language generation (NLG) applications, particularly for open-ended language generation tasks such as story generation, namely generating a reasonable story from a prompt or a leading context. While existing generation models (Fan et al., 2018; Radford et al., 2019) can generate texts with good intra-sentence coherence, it is still difficult to plan a coherent plot throughout the text, even when using the powerful pretrained models, as illustrated in Figure 1. Pretrained generation models have shown state-of-the-art performance on various NLG tasks such as summarization and translation (Radford et al., 2019; Lewis et al., 2020). However, such tasks  Corresponding author  Human Writer: A: He thought that would change if he joined a sport. Leading Context: Tim wasn't very athletic. BART: A: He wasn't good at basketball . B: He was a part of the basketball team . C: Tim was offered a job at a local basketball team . D: Tim played baseball better in the city. provide sufficient source information in the input for generating desired texts, while open-ended generation tasks require expanding reasonable plots from very limited input information (Guan et al., 2020). As exemplified in Figure 1, we observe severe issues of incoherence when applying BART for story generation. Although BART performs reasonably well at generating some concepts related to the context (e.g., basketball, player ), they are used incoherently in the generated texts, which is manifested in repetitive plots (e.g., the sentences B and C ), unrelated events (e.g., played baseball better ) and conflicting logic (e.g., not good at basketball but in the basketball team ). These issues are also commonly observed in other NLG models (Holtzman et al., 2020; Guan and Huang, 2020). We argue that existing models are rarely trained beyond the token-level co-occurrence, and therefore they can easily generate related concepts but do not arrange them reasonably. In contrast, human writers always first fully understand the semantics (e.g., some key events such as try out, not make the cut ) and the discourse relations (e.g., temporal orders) among the already written sentences before deciding the following content. In this way, the writers can write coherent stories even with few related concepts, as shown in Figure 1. Therefore, it is important for subsequent generation to capture high-level features in the context. In this paper, we propose HINT , a generation model equipped with HI gh-level representations for lo N g T ext generation . Typical generative models usually train a left-to-right decoder by next word prediction based on the attention to all the prefix words. In order to encourage the model to capture high-level features, we extend the decoder to represent the prefix information at sentence level and discourse level, respectively, with special tokens which are inserted at the end of each sentence. To effectively learn the representations, we propose two pretraining objectives including: (a) semantic similarity prediction , which requires predicting the inter-sentence similarity using the sentence-level representation, with the powerful sentence understanding model SentenceBERT (Reimers and Gurevych, 2019) as the teacher model; and (b) sentence order discrimination , which requires distinguishing between the normal and shuffled sentence orders using the discourse-level representation. The objectives are designed to help the decoder capture the semantics and discourse structure of the prefix, which can benefit modeling the long-range coherence when generating long texts. We summarize our contributions in two folds: I. We propose a generation model named HINT for long text generation. HINT derives high-level representations for each decoded sentence to model the long-range coherence. We adopt two pretraining objectives called similarity prediction and order discrimination to learn the representations at sentence level and discourse level, respectively. II. We conduct extensive experiments on commonsense story and fiction generation tasks. Results show that HINT can learn meaningful high-level representations and generate more coherent long texts than baselines. Long Text Generation Recent studies tackle the incoherence problem in long text generation from the following perspectives. Li et al. (2015) adopted a hierarchical RNN-based decoder to learn the sentence representation but without any external supervision. Shao et al. (2017) proposed a self-attention mechanism to attend on the prefix by appending it to the RNN-based encoder, which is a similar idea with the vanilla Transformer (Vaswani et al., 2017). However, the token-level self-attention mechanism still struggles to model high-level dependency in the context. Recent works proposed several multi-step generation models (Fan et al., 2018; Yao et al., 2019; Shao et al., 2019; Tan et al., 2020; Goldfarb-Tarrant et al., 2020), which first plan high-level sketches and then generate texts from the sketches. However, the lack of exposure to degenerate sketches may impair the generation performance since the models are only trained on sketches constructed from golden truth texts (Tan et al., 2020). Another line is to incorporate external knowledge into generation especially for commonsense story generation (Guan et al., 2020; Xu et al., 2020). However, the methods may not be always effective for other types of generation tasks. Guan et al. (2020) also required the decoder to distinguish true texts from negative samples to alleviate potential issues such as repetition. But the classification objective does not provide explicit guidance for generation at each step. Therefore, the coherence of language generation is still an open problem. High-Level Language Representation Signifi-cant advances have been witnessed in many NLP tasks with pretrained contextualized representation (Peters et al., 2018; Devlin et al., 2019). However, most models were limited on token-level representation learning, which is not enough for capturing the hierarchical structure of natural language texts (Ribeiro et al., 2020). Several works have tried to learn high-level representation. Skip-Thought vectors (Kiros et al., 2015) learned to encode a sentence by reconstructing its neighboring sentences. HLSTM (Yang et al., 2016) considered a 1 The codes are available at https://github.com/ thu-coai/HINT HINT Decoder A B sen dis C D sen dis E F                       sen dis   Sentence-Level Representations Discourse-Level Representations Task 2: Similarity Prediction Task 3: Order Discrimination HumanWritten Texts NegativeSamples Sentence 1 Sentence 2 Sentence 3 Feed Forward All Hidden Representations Task 1: Language Modeling ... ... hierarchical LSTM-based encoder to learn the contextualized sentence representation by downstream classification. HIBERT (Zhang et al., 2019) incorporated the hierarchical architecture to BERT (De-vlin et al., 2019) and learned sentence representation by recovering masked sentences. SentenceBERT (Reimers and Gurevych, 2019) derived sentence representation by fine-tuning BERT for natural language inference. CONPONO (Iter et al., 2020) and SLM (Lee et al., 2020) further trained BERT to understand relations among sentences at discourse level by distance prediction and sentence unshuffling, respectively. However, all these models focused on enhancing the representation of encoders for language understanding, while improving decoders by high-level representation for long text generation is yet to be well investigated. Our task can be defined as follows: given an input X = ( x 1 , x 2 ,    , x m ) (e.g., a beginning or a prompt), the model should generate a multi-sentence text Y = ( y 1 , y 2 ,    , y n ) with a coherent plot (each x i or y i is a token). To tackle the problem, the conventional generation models such as BART commonly employ a bidirectional encoder and a left-to-right decoder to minimize the negative log-likelihood LLM of human-written texts: LLM =  n (cid:88) t =1 log P ( y t | y <t , X ) , (1) P ( y t | y <t , X ) = softmax ( H t W + b ) , (2) H t = Decoder ( y <t , { S i } mi =1 ) , (3) { S i } mi =1 = Encoder ( X ) , (4) where H t is the decoder's hidden state at the t -th position computed from the context (i.e., the prefix y <t and the input X ), and S i is the contextualized representation of x i acquired from the encoder, W and b are trainable parameters. However, as aforementioned, the models often generate incoherent texts due to the decoder's inability to capture high-level features of the prefix sentences. Therefore, we extend the decoder with high-level representations to gather the prefix information. Specifically, we split the human-written texts into sequential sentences and add special tokens at the end of each sentence, which will be used to aggregate their respective semantics and their discourse relations with one another during decoding. To this end, we devise two pretraining tasks besides the standard language modeling objective, including similarity prediction and order discrimination to learn the sentence-level and discourse-level representations, respectively, as Figure 2 shows. Although we only consider sentence as segments in this work, our method can be easily extended to other syntactic levels such as phrases or paragraphs. 3.2 Sentence-Level Representation Assume that the target text Y consists of K sentences, denoted from Y 1 to YK (e.g., AB and CD in Figure 2). We insert a special sentence token, (cid:104) sen (cid:105) , at the end of every sentence in Y , which is designed to aggregate the semantics of each sentence. Let H s k (1 (cid:54) k (cid:54) K ) denote the decoder's hidden state at the position where the k -th sentence token is the golden truth for next token prediction. We expect H s k to be a meaningful sentence representation for Y k , which means semantically similar sentences have close representations in the vector space. Since sentence representation has been well studied for language understanding with many powerful models such as SentenceBERT (Reimers and Gurevych, 2019), we propose to directly transfer their semantic knowledge for our sentence representation learning. Specifically, we require the HINT decoder to predict the similarity of any two sentences Y i and Y j only using the corresponding sentence representations H s i and H s j , with the SentenceBERT similarity as the golden truth 2 . We do not directly learn the SentenceBERT representation for each sentence but the similarity score to avoid the discrepancy between different model bias. Furthermore, to alleviate the innate bias of SentenceBERT, we do not enforce HINT to exactly fit the golden similarity. Instead, it would be enough that the difference between the predicted score and the golden similarity is less than a margin   [0 , 1] . Formally, the loss function L Sen for the similarity prediction task can be derived as follows: L Sen = 1 K 2 K (cid:88) i =1 K (cid:88) j =1 max ( | p ij  t ij | , ) , (5) p ij = sigmoid ( s ij + s ji ) , (6) s ij = ( H s i ) TW s H s j , (7) where t ij is the golden similarity, p ij is the predicted similarity score, s ij is an intermediate variable to guarantee p ij is symmetric with respect to i and j , W s is a trainable parameter to transform the representation space of HINT to that of SentenceBERT. The task explicitly exerts external supervision to learn the sentence-level representation, enhancing the ability of the HINT decoder to fully understand the semantics of prefix sentences. 2 The SentenceBERT similarity is computed as the cosine distance of two sentence embeddings which are derived by applying mean-pooling on the output vectors of SentenceBERT. And we normalize the results to [0 , 1] range by linear scaling. In analogy to the sentence-level representation learning, we also insert a special discourse token, (cid:104) dis (cid:105) , after every sentence and the corresponding sentence token to gather the discourse information between different sentences. Let H d k (1 (cid:54) k (cid:54) K ) denote the decoder's hidden state at the position where the k -th discourse token is the golden truth to be predicted. H d k should be a meaningful representation which can be used to derive discourse relations with others (e.g., the k -th sentence precedes another one in terms of the temporal order). Previous work has shown that reconstructing the correct order from shuffled sentences helps understand the discourse relations (Lee et al., 2020). However, the unshuffling task is not directly applicable for NLG since the decoder should learn to dynamically model the discourse structure in the decoding process rather than wait until finishing decoding the whole text. Therefore, we propose to learn the discourse-level representation in a pair-wise manner by discriminating whether the order of two sentences is correct. Formally, we minimize the cross-entropy loss L Dis as follows: L Dis = 2 K ( K  1) K (cid:88) i =1 K (cid:88) j>i l ij , (8) l ij =  o ij log q ij  (1  o ij ) log (1  q ij ) (9) q ij = sigmoid (cid:0) ( H d i ) TW d H d j (cid:1) , (10) where o ij is the golden label (1 if Y i should precede Y j , 0 otherwise), q ij is the predicted discrimination score, and W d is a trainable parameter. Compared with the sentence-level representation H s k which aggregates the semantics of a single sentence, the discourse-level representation H d k focuses more on the relationship with other sentences, thereby improving HINT 's ability to capture the high-level features in both content and order. To learn the high-level representations more effectively, we propose to augment the training corpus by automatically constructing negative samples from the human-written texts for pretraining. Specifically, for the order discrimination task, we randomly shuffle the sentences in human-written texts as negative samples. And for the similarity prediction task, besides the negative samples with shuffled sentences, we also randomly repeat a sentence, or substitute a sentence with another from other texts as negative samples. We expect the negative samples to help enhance the generalization ability of HINT during fine-tuning or inference. In summary, the overall loss function L Pre for pretraining is computed as follows: L Pre = LLM +  1 L Dis +  2 L Sen , (11) where we optimize the language modeling objective LLM only on the human-written texts, L Dis on the human-written texts and the negative samples with shuffled sentences, and L Sen on all the human-written texts and the negative samples. 1 and  2 are adjustable scale factors. By pretraining with the proposed two objectives, the decoder can better capture the semantics and discourse structures in the context. And during fine-tuning, we train HINT only with the language modeling objective. Since our approach can adapt to all the generation models with auto-regressive decoders (e.g., GPT-2 (Radford et al., 2019), UniLM (Dong et al., 2019), etc.), we use BART as the base framework of HINT , which has been shown to have strong performance for long text generation (Goldfarb-Tarrant et al., 2020). And we also provide the performance of GPT-2 widely used in the literature. Due to the limited computational resources, we follow BARTBASE 's hyper-parameters and utilize the public pretrained checkpoint to initialize HINT . The batch size is set to 10 and the maximum sequence length is set to 512 for both the encoder and the decoder. The margin  in Equation 5 is set to 0.1 and we present the results with other settings of  in the appendix. Both the scale factors  1 and  2 in Equation 11 are set to 0.1. We adopt BookCorpus (Zhu et al., 2015) as our pretraining dataset and split each text to sentences using NLTK (Bird and Loper, 2004). We create the training texts by taking a sentence as the input and the following ten sentences as the target output. Besides, we construct the same number of negative samples with the human-written texts. And it is evenly possible for a negative sample to be repeated, substituted or shuffled. We pretrain HINT on BookCorpus for 0.1M steps. We evaluate HINT on ROCStories ( ROC for short) (Mostafazadeh et al., 2016) and Writing-Prompts Writing-Prompts ( WP for short) (Fan et al., 2018). ROC contains 98,162 five-sentence commonsense stories. We follow Guan et al. (2020) to delexicalize stories in ROC by masking all the names with special placeholders to achieve better generalization. WP originally contains 303,358 stories paired with writing prompts, which are usually unconstrained on writing topics. Considering that using too many examples for fine-tuning may weaken the influence of post-training, we randomly selected stories from the original validation set and test set of WP for the subsequent experiments. We regard the first sentence and the prompt as the input to generate a text for ROC and WP, respectively. And we only retain the first ten sentences (split using NLTK) of the texts in WP for fine-tuning. We present more details in Table 1. The batch size is set to 10/4 for ROC/WP, respectively. And other hyper-parameters are the same as the pretraining phase. We compared HINT with the following baselines: Seq2Seq: It generates a text conditioned upon the input. For better performance, We implement the baseline by training BART from scratch on the downstream datasets without pretraining. Plan&Write: It first plans a keyword sequence conditioned upon the input; and then generates a text based on the keywords (Yao et al., 2019). We implement the model based on the codes provided by the original paper. GPT-2 and BART: They are fine-tuned on the downstream datasets with the language modeling objective. BART-Post: It is first post-trained on the pretraining dataset with the original pretraining objectives of BART (text infilling and sentence permutation) for the same number of steps with HINT ; and then fine-tuned on the downstream datasets with the language modeling objective. BART-MTL: The model is trained by fine-tuning BART on the downstream datasets with multi-task learning (MTL), including the language modeling objective and an auxiliary multi-label classification objective (Guan et al., 2020), which requires distinguishing human-written texts from auto-constructed negative samples. Furthermore, we conduct ablation tests by removing the proposed components respectively to investigate the influence of each component. Besides, we also demonstrate the adaption of our approach to general language generation models by directly fine-tuning BART and HINT on downstream datasets with the proposed two objectives as auxiliary tasks. For fair comparison, we set all the pretrained models to the base version. And we also insert the sentence token and discourse token into each training text for all the baselines. We generate texts using nucleus sampling (Holtz-man et al., 2020) with p=0.9 and a softmax temperature of 0.7 (Goodfellow et al., 2016) to balance the trade-off between diversity and fluency. And we set the probability of generating (cid:104) dis (cid:105) to 1 if the last token is (cid:104) sen (cid:105) to ensure that HINT can obtain the high-level representations for each sentence. And during evaluation, we remove the special tokens in the generated texts. We apply these settings to all the baselines. Evaluation Metrics We adopt the following automatic metrics to evaluate the performance on the test sets: (1) Perplexity (PPL) : Smaller perplexity scores indicate better fluency in general. We do not count the probability values at the positions where the sentence or discourse token is the golden truth. (2) BLEU (B-n) : We use n = 1 , 2 to evaluate n -gram overlap between generated texts and human-written texts (Papineni et al., 2002). (3) Lexical Repetition (LR-n) : The metric computes the percentage of those texts which repeat a 4-gram at least n times in all the generated texts (Shao et al., 2019). We set n = 2 for ROC and n = 5 for WP. (4) Semantic Repetition (SR-n) : The metric first computes the average topn SentenceBERT similarity between any two sentences in each generated text, and then averages the results as the final score. We set n = 1 for ROC and n = 10 for WP. (5) Distinct-4 (D-4) (Li et al., 2016): We adopt distinct-4, the ratio of distinct 4-grams to all the generated 4-grams, to measure the generation diversity. (6) Context Relatedness: It is a learnable automatic metric (Guan and Huang, 2020). First, we train a classifier with RoBERTa BASE (Liu et al., 2019) to distinguish human-written texts and negative samples constructed by substituting words, phrases and sentences of human-written texts randomly. Then, we use the average classifier score of all the generated texts to measure the context relatedness. (7) Sentence Orders: In analogy to relatedness measurement, we train another classifier to distinguish human-written texts and negative samples where sentences are randomly shuffled. We use the average classifier score to measure sentence orders. We train the last two metrics based on the training sets of the downstream datasets. Results on ROC We show the results on ROC in Table 2. We do not provide the perplexity scores of Plan&Write and GPT-2 since they do not tokenize texts with the same vocabulary as used in BART. HINT outperforms all the baselines in terms of perplexity, indicating the better ability to model the texts in the test set. And HINT can generate more word overlaps with reference texts as shown by better BLEU scores. It is accordant with the previous observation (Xu et al., 2020) that Plan&Write has less lexical repetition than pretraining models possibly because small models are better at learning short term statistics (e.g., n -gram) but not long term dependencies. However, HINT improves the situation compared with GPT-2 and BART, and has less semantic repetition than all the baselines, indicating the better ability of HINT to capture semantic features. Besides, our approach does no harm to the generation diversity. HINT also outperforms baseline models in generating related events and arranging a proper order, as shown by the higher relatedness and order scores. Furthermore, fine-tuning with the proposed objectives as auxiliary tasks can further reduce the lexical and semantic repetition, and improve the relatedness and order scores for both BART and HINT , suggesting the general benefit of modeling the long-range coherence at sentence level and discourse level. Besides, the ablation test shows that the sentence-level and discourse-level representations are relatively more important to enhance the ability to generate texts with related events and reasonable orders, respectively. And both of them contribute to reducing semantic redundancy. When post-training only with the language modeling objective, almost all the metrics drops substantially, indicating the importance to model high-level coherence. golden texts. We summarize the possible reasons as follows: (a) It is still difficult for the learned classifier to judge implicit relatedness in some golden texts, which may require a strong reasoning ability. (b) There exist some noisy texts with poor relatedness in the golden texts. And (c) the systems tend to generate a limited set of texts (as demonstrated by much lower distinct-4 than golden texts) with generic plots (Guan et al., 2020), which may get high relatedness scores easily. However, we believe the learnable metric is still meaningful to compare different models with similar diversity regarding the context relatedness. Results on WP We present the results on WP in Table 3. We use a larger n to compute the lexi-cal/semantic repetition since we find that all the models tend to repeat similar texts easily when generating texts with hundreds of words. And we do not provide the relatedness and order scores because it is difficult to train satisfactory classifiers to distinguish human-written texts from negative samples well. Table 3 shows that HINT outperforms baselines except for lexical repetition, which is accordant with the results on ROC. Therefore, the high-level representations are effective for generating long texts with different lengths and domains. For manual evaluation, we conduct pair-wise comparisons with two strong baseline models (BART and BART-Post), and three ablated models of HINT We randomly sample 200 texts from the test set of Models PPL  B-1  B-2  LR-5  SR-10  D-4  Seq2Seq 129.51 0.165 0.070 0.623 0.819 0.283 Plan&Write N/A 0.199 0.070 0.524 0.851 0.272 GPT-2 N/A 0.200 0.073 0.655 0.883 0.287 BART 34.42 0.205 0.075 0.620 0.854 0.291 BART-MTL 35.71 0.198 0.076 0.654 0.846 0.305 BART-Post 35.11 0.205 0.076 0.671 0.862 0.271 HINT 32.73 0.224 0.084 0.567 0.805 0.313 w/oSen 33.08 0.216 0.080 0.598 0.823 0.303 w/oDis 33.18 0.223 0.083 0.588 0.818 0.307 w/oSen&Dis 33.71 0.207 0.076 0.610 0.845 0.280 GoldenText N/A N/A N/A 0.007 0.448 0.928 Table 3: Automatic evaluation results on WP. For each pair of texts (one by our model and the other by a baseline, along with the input), three annotators are hired to give a preference (win, lose, or tie) in terms of fluency and coherence, respectively. We adopt majority voting to make final decisions among the three annotators. We resort to Amazon Mechanical Turk (AMT) for annotation. We follow Xu et al. (2020) to define fluency as a measure of intra-sentence linguistic quality and grammatical correctness, and coherence as inter-sentence relatedness, causal and temporal dependencies. Note that the two aspects are independently evaluated. Besides, we control the annotation quality by fil-tering out those annotations where the annotator can not make reasonable judgments when comparing a human-written text with a negative sample. Furthermore, we also ask workers to annotate the specific errors in the generated texts. We show the annotation instruction and the error analysis of different models in the appendix. Table 4 shows the manual evaluation results. All the results show moderate inter-annotator agreement (0.4 (cid:54)  (cid:54) 0.6) or substantial agreement (0.6 (cid:54)  (cid:54) 0.8). And we can see that HINT performs significantly better than baselines in coherence by capturing the high-level features, and has comparable fluency with baselines. It is still necessary to further investigate whether the learned representations help HINT capture the high-level coherence better. Therefore, we propose to evaluate the models using individual language modeling tests in different aspects (Ribeiro et al., 2020). To this end, we construct coherent and inco-too long texts. herent examples based on the test set of ROC, and compute perplexity on the examples of different aspects. Specifically, we focus on the following aspects: semantic repetition, relatedness, negation, causal and temporal relationship. We select human-written texts as coherent examples and construct incoherent examples by perturbing human-written texts. For example, we select those texts with time-related words (e.g., then ) as coherent examples for testing in the temporal relationship. And we exchange two sequential events connected by then of a human-written text or substitute before with after as incoherent examples of the aspect. We show more details in the appendix. We present the results in Table 5. HINT can model the context coherence better in the above aspects than baseline models (lower perplexity on the coherent examples), and recognize the incoherent errors more effectively (higher perplexity on the incoherent examples). By contrast, both BART-Post and HINT (w/o Sen&Dis) achieve an overall drop of perplexity compared with BART even on the negative examples, indicating that they may still focus on capturing the token-level features . As for the ablation study, we can see that the sentence-level representation enhances the ability of HINT to capture the relatedness, negation and semantic repetition, while the discourse-level representation works mainly for causal and temporal relationship. However, we also notice the insignificant improvement of HINT compared with BART in recognizing the unreasonable causal and temporal relationship, which may require injecting explicit inferential knowledge besides learning sentence orders. We present several cases in the appendix to demonstrate that HINT can derive meaningful sentence-level and discourse-level representations, and generate texts with better coherence than baselines with the help of the representations. We present HINT , a generation model for ation, which can represent the prefix information at sentence level and discourse level in the decoding process. We propose two pretraining objectives including inter-sentence similarity prediction and sentence order discrimination to learn the sentence-level and discourse-level representations, respectively. Extensive experiments demonstrate that HINT can generate more coherent texts with related context and proper sentence orders than strong baselines. Further analysis shows that HINT has better ability of language modeling thanks to ability of modeling high-level coherence. This work was supported by National Key R&D Program of China, under Grant No. 2020AAA0104500. This work was jointly supported by the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096), and the Guoqiang Institute of Tsinghua University, with Grant No. 2019GQG1 and 2020GQG0005. We would also like to thank the anonymous reviewers for their invaluable suggestions and feedback. We conduct the experiments based on two existing public datasets ROCStories and WritingPrompts, which are widely used for commonsense story generation and fiction generation tasks, respectively. Automatic and manual evaluation show that our model outperforms existing state-of-the-art models on both datasets, suggesting the generalization of our model to different domains. Besides, our approach can be easily extended to different syntactic levels (e.g., phrase-level, paragraph-level), different model architectures (e.g., GPT, UniLM) and different generation tasks (e.g., dialog generation, essay generation). We resorted to Amazon Mechanical Turk (AMT) for manual evaluation. We did not ask about personal privacy or collect personal information of annotators in the annotation process. We hired three annotators and payed each annotator $0.05 for comparing each pair of stories. The payment is reasonable considering that it would cost average 30 seconds for an annotator to finish a comparison.
abstain abstain abstain objective objective result abstain other abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain abstain objective abstain objective objective abstain abstain abstain objective objective abstain objective abstain method abstain other other other other other other other other other other other other other other other other other other other other other method other abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain result abstain abstain method objective abstain abstain other other other method abstain abstain abstain method method abstain