Story generation is an open-ended and subjective task, which poses a challenge for evaluating story generation models. We present CHOOSE YOUR OWNADVENTURE , a collaborative writing setup for pairwise model evaluation. Two models generate suggestions to people as they write a short story; we ask writers to choose one of the two suggestions, and we observe which model's suggestions they prefer. The setup also allows further analysis based on the revisions people make to the suggestions. We show that these measures, combined with automatic metrics, provide an informative picture of the models' performance, both in cases where the differences in generation methods are small (nucleus vs. topk sampling) and large (GPT2 vs. Fusion models). Systems that automatically generate text suggestions to human authors have emerged as a new application of natural language generation models. Evaluating such models, however, is challenging. Typically, writers rate a single system's quality after some period of use, for example while authoring an entire story or poem (e.g., Clark et al., 2018; Ghazvininejad et al., 2017). A model's quality is measured using Likert scale scores, sometimes combined with additional analysis, like the type or quantity of writer edits (e.g., Roemmele and Gordon, 2015; Akoury et al., 2020). In contrast, a pairwise system evaluation where evaluators are given two suggestions at the same time and asked to choose between them would allow researchers to compare generation models directly. Comparative evaluations have been shown to produce more reliable and consistent results than Likert-scale ratings (Callison-Burch et al., 2007; Kiritchenko and Mohammad, 2017), and they have been used to evaluate natural language generation systems for translation and dialogue (Otani et al., 2016; Sedoc et al., 2019). We propose CHOOSE YOUR OWNADVENTURE (CYOA), a protocol for pairwise evaluations of collaborative writing models, focusing on story generation. Instead of scoring a single model, we compare two models. At fixed points during the writing process, each generates a suggestion, and writers choose one to continue their story (see Fig. 1). The result is utterance-level feedback on which model's generated text writers prefer at that point in the story. Along with the writer's revisions to the generated suggestions and comparisons between the generated and human-authored portions of the story, this evidence can help a researcher answer the following questions about their model: to human-authored text? In this paper, we show how CYOA can answer these questions and provide insights into story model behavior, both in cases when the expected differences in text quality are large (e.g., the text is generated with two different models; 3) and when they are small (e.g., the text is generated with the same model but using two different sampling methods; 4). CYOA allows human and automatic evaluations to be collected simultaneously; we run standard automatic evaluations of text quality on the collaboratively-generated text and get results consistent with previous analyses of statically-generated text. CYOA is useful to both NLG researchers and story writers; writers report being happy with the stories they write with the system and that the paired suggestions help them come up with new ideas. We release a template website for CYOA and the evaluation script 1 to support future story and collaborative writing evaluation work. OWNADVENTURECYOA evaluates a pair of story generation models by having people select and interact with text generated by each of the models as they write a story. Both models generate suggestions for the writer at the same point in the story, and the writer must choose between the two suggestions, forcing a pairwise comparison of the two models. By having multiple people write stories with the two models, we can aggregate their preferences and interactions with the suggestions and analyze them to provide feedback on the two models. To allow the writers control over the story while still encouraging them to use the suggestions, CYOA uses a turn-taking writing process, with writers alternating between writing by themselves and then receiving suggestions to continue the story (Swanson and Gordon, 2012; Clark et al., 2018). The writer begins the story by writing the first sentence alone; an image (Fig. 2 in App. A) is provided as an optional prompt to help them get started. Once the writer submits the writing from their turn, two models each generate a suggestion to continue the story, which are presented to the writer in random order. As shown in Fig. 1, the 1 github.com/eaclark07/cyoa writer then chooses which of the suggestions they prefer and edits it as they wish before adding it to the story. It is then the writer's turn to write alone again. This process repeats 5 times, at which point the story is finished and submitted. Each turn in the story has to be between 20 and 260 characters for it to be submitted to the story. Other than length, there is no restriction on how writers can edit the suggestions; they can delete the suggestion entirely or submit it as-is. When editing a computer-generated suggestion, the writer can change their mind and select the other model's suggestion instead, but once a writer submits a turn, they cannot go back to edit it later. After the finished story is submitted, participants are asked Likert-scale and open-ended questions about the system and the suggestions they received. We asked participants to indicate on a 5-point Likert scale (ranging from Strongly Disagree to Strongly Agree) how much they agreed with the following statements:  I'm happy with my final story. I felt the system and I were working collaboratively to write the story. I thought having the suggestions was useful while writing the story. The suggestions connected to what had happened in the story so far. The suggestions helped me come up with new ideas. What were you looking for in the suggestions? We chose these questions for this project to capture people's reactions to the overall writing setup and a general sense of areas for improving story generation models. However, these questions could be eliminated or adjusted to fit the evaluation goals of the researcher. A demo of CYOA is at homes. cs.washington.edu/~eaclark7/ multi-model-demo . From the writing setup, we collect the generated suggestions from each model, the writers' preferences between the two models, and the revisions they make to the generated text. We analyze these sources of information to answer three questions NLP practitioners have when evaluating their models. There are many analyses researchers could run with the data gathered from CYOA beyond those listed here; we include some examples. (Q1) Is my model better at generating story suggestions than a baseline model? CYOA reports how many of the model's suggestions people chose to work with vs. the baseline's suggestions. We further break this down by the suggestion round (15) to see if the writers' preferences change over the course of the story. Another option would be to break down the writers' preferences by writer attributes, e.g., to analyze the effect of the author on the stories or desired suggestions (August et al., 2020). (Q2) How useful are the models' suggestions? We analyze the revisions writers make to the suggestions to see how much of the generated text they find useful for continuing their story. We use three metrics to see how much of the original text is preserved after a writer's revisions. Levenshtein edit distance measures the number of character insertions, deletions, and substitutions the writers made, and Jaccard similarity measures the proportion of tokens that are shared between the original and the edited text. User Story Edit Ratings (USER; Akoury et al., 2020) 2 measures similarity by recursively counting the longest contiguous substrings between the edited and the original text. These edit-based metrics capture exact matches between the texts, measuring how much of the generated content makes it to the final story in the strictest sense. However, other metrics could be used if the researcher is interested in capturing broader notions of similarity, e.g., embedding-based measures like cosine similarity or BERTScore (Zhang et al., 2020). (Q3) How do the models' generated texts compare to human-authored text? Pairwise comparison gives us the models' relative quality; comparing them to human-authored text gives an idea of their absolute quality. To do this, we take the parts of the story the writer wrote alone (i.e., the turns without generated suggestions) and compare it to the generated text. We look at average sentence length (a common proxy for text complexity in stories; See et al., 2019; Roemmele et al., 2017) and distinctn , a measure of repetition (Li et al., 2016). As in See et al. (2019), we also look the concreteness of the text's nouns and verbs, us-2 github.com/dojoteef/storium-frontend ing the concreteness ratings from Brysbaert et al. (2014). If the system is being used to evaluate a model that focuses on a specific aspect of stories, e.g., events or characters, this analysis could be extended to compare how these specific elements are introduced and referenced in the machine-generated vs. human-authored text. We first test CYOA with two popular story generation models: (1) FUSION , the fusion model from Fan et al. (2018), which uses a fusion mechanism to combine two convolutional sequence-to-sequence models; and (2) GPT 2, the small GPT2 model (Rad-ford et al., 2019) finetuned on story data and using topk sampling (Fan et al., 2018). We compare FUSION and GPT 2 to see how CYOA can evaluate two models with different underlying architectures; they are also both common story generation baselines (See et al., 2019; Xu et al., 2020; Rashkin et al., 2020). To train the models, we use the WritingPrompts dataset (Fan et al., 2018), a collection of writing prompts from Reddit paired with stories. During the CYOA evaluation, both models generate their suggestions conditioned on the whole story written so far. (Data and model details in App. B and C.) We run CYOA on Amazon Mechanical Turk with 105 Turkers to compare the two models. Each Turker can only complete the task once. Turkers are required to have over 1,000 tasks approved, have an 95% approval rate, and be from the United States, and they are paid $2.50 for participating in the study. The study was approved by our instiu-tion's Institutional Review Board. (Q1) Table 1 shows that, of the 525 suggestion pairs, Turkers significantly 4 preferred the GPT 2 suggestions over FUSION , choosing them 65.7% of the time. Breaking it down by suggestion round 1 5, the writers' preference for the GPT 2 was largest at the beginning of the story and decreased over the course of the story. To understand why, we look at how writers edited the suggestions and how the generated text compared to human-authored text. (Q2) In Table 2, all three edit metrics show that writers used significantly 5 more of the accepted GPT 2 suggestion text in their story than the accepted FUSION suggestion text. When we break down the scores by round, we see that this is true regardless of where in writer is in the story (see Table 7 in App. D.1). Taken with the pairwise results, this points to GPT 2 as the better collaborative story generation model. FUSION , perhaps due to its hierarchical structure, did not generate as many useful suggestions as GPT 2 in the interactive setting. (Q3) Finally, we look at how the generated text compares to the story text the writers wrote alone. From Table 3, we see that GPT 2 generates shorter, more concrete, and more repetitive suggestions than FUSION . Both models generate shorter sentences than people, and GPT 2 generates more concrete nouns and verbs than FUSION , corroborating the analysis of See et al. (2019). GPT 2 generated the most repetitive text, which may explain why it is chosen less frequently as the story goes on. FUSION 's subhuman level of repetition indicates it often fails to refer back to the story context, as illustrated by the low Likert-scale scores for The suggestions connected to what had happened in the story so far. (Fig. 3 in App. D.2). Our second experiment compares text generated from GPT 2 but now using different sampling strategies: TOP-K (as in 3) and NUCLEUS sampling (Holtzman et al., 2020). (Model details in App. C.) Here we expect to see narrower differences in the generated text than we did in 3. Comparing TOP-K vs. NUCLEUS focuses on CYO A's ability to compare models with fine-grained differences. 103 5 Mann-Whitney U test: p < 0 . (Q1) Table 4 shows Turkers preferred the TOPK suggestions over the NUCLEUS suggestions for 53.4% of the 515 suggestion pairs writers received; as expected, a smaller difference than in 3 and not significant. 7 Again, the writers' preference for TOP-K decreased over the course of the story, with NUCLEUS slightly more popular by the end. (Q2) In Table 5, all three metrics show that writers used more of the NUCLEUS -sampled text than the TOP-K -sampled text, though the difference is not significant. 8 Despite writers' slight preference for TOP-K -sampled suggestions, when they choose NUCLEUS -sampled suggestions, they preserve more of the generated text. Table 8 (App. D.1) shows that difference is largest at the beginning and end of the story. This suggests TOP-K 's safer suggestions may be less useful, especially when starting or finishing the task. to the same requirements. 7 Binomial test: p = 0 . 07 . 8 Mann-Whitney U test: p = 0 . 19 (ED), p = 0 . 23 (JS), and p = 0 . 27 (USER). (Q3) Table 6 shows that TOP-K -generated text is shorter, more concrete, and more repetitive than NUCLEUS -generated text. NUCLEUS 's text comes closer to human-levels of repetition, consistent with the findings of Holtzman et al. (2020) and Akoury et al. (2020). CYOA benefits writers as well as researchers. The results of the writer feedback across both experiments indicate that writers enjoy the paired-suggestion writing experience, regardless of which models they wrote with. The Likert-scale responses were particularly positive for I'm happy with my final story. ( FUSION vs. GPT 2: mean = 3 . 83 , NUCLEUS vs. TOP-K : mean = 3 . 84 ) and The suggestions helped me come up with new ideas. ( FUSION vs. GPT 2: mean = 3 . 80 , NUCLEUS vs. TOP-K : mean = 3 . 79 ). This compares favorably to single-suggestion collaborative story writing systems that use a similar writing process; Clark et al. (2018) report writers gave a mean score of 3.28 9 for happiness with the story they wrote with their collaborative writing system. Full Likert-scale results are in App. D.2. The positive reactions from participants indicate this format could work well on alternative crowdsourcing platforms, like LabintheWild, 10 or launched as an independent writing game, similar to Akoury et al. (2020). Collaborative writing systems have been developed in domains like poetry (Ghazvininejad et al., 2017), slogans (Clark et al., 2018), and stories (Roemmele and Gordon, 2015; Goldfarb-Tarrant et al., 2019; Akoury et al., 2020). Like Storium (Akoury et al., 2020), we focus on the potential to use these systems as evaluation platforms. However, we suggest using paired suggestions in collaborative writing systems to directly compare generation models. ChatEval (Sedoc et al., 2019) collects human evaluations for paired chatbot utterances and Otani et al. (2016) for paired translations, but the generated text is static. By having writers interact with dynamically generated suggestions, collaborative writing systems reward helpful and robust generation models, underemphasized attributes in current evaluations (Zellers et al., 2021; Ethayarajh and Jurafsky, 2020). CYOA allows researchers to collect human and automatic evaluations for story generation models in a single collaborative writing task. The paired suggestions allow direct comparisons between two models, and automatic-metric comparisons among generated text, its revisions, and the human-authored portions provide additional insight. We expect CYOA evaluations to accelerate progress on applications for collaborative writing between humans and machines. This research was supported in part by a NSF graduate research fellowship and the DARPA CwC program through ARO (W911NF-15-1-0543). The authors would also like to thank the ARK group, Ari Holtzman, Nader Akoury, and Yejin Choi for their help and feedback, the reviewers for their helpful comments, and the participants who took part in our study.
abstain method result abstain result abstain abstain abstain abstain abstain abstain objective method abstain abstain abstain abstain result result abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method method method abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method method method method abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other method method other other abstain abstain abstain other other