People often share personal narratives in order to seek advice from others. To properly infer the narrator's intention, one needs to apply a certain degree of common sense and social intuition. To test the capabilities of NLP systems to recover such intuition, we introduce the new task of inferring what is the advice-seeking goal behind a personal narrative. We formulate this as a cloze test, where the goal is to identify which of two advice-seeking questions was removed from a given narrative. The main challenge in constructing this task is finding pairs of semantically plausible advice-seeking questions for given narratives. To address this challenge, we devise a method that exploits commonalities in experiences people share online to automatically extract pairs of questions that are appropriate candidates for the cloze task. This results in a dataset of over 20,000 personal narratives, each matched with a pair of related advice-seeking questions: one actually intended by the narrator, and the other one not. The dataset covers a very broad array of human experiences, from dating, to career options, to stolen iPads. We use human annotation to determine the degree to which the task relies on common sense and social intuition in addition to a semantic understanding of the narrative. By introducing several baselines for this new task we demonstrate its feasibility and identify avenues for better modeling the intention of the narrator. Computers are useless. They can only give you answers. Pablo Picasso People often share their personal experiences to elicit advice from others. These personal narratives provide the necessary context for properly understanding the informational goals of the narrators. Endowing automated systems with the capability to infer these advice-seeking intentions Personal narrative : I am generally a person who needs a lot of sleep, but today I was not able to sleep more than 6 hours and I am extremely tired. My eyes hurt and two hours later I have programming [lesson] so I have to be alert. I've already drunk a cup of coffee and although I rarely drink coffee, it had no effect on me. I am not at home so I have limited possibilities as for food. I don't want to do anything too unhealthy such as drinking 10 cups of coffee, tho I may consider drinking another one. Which advice-seeking question is more likely to have been asked by the narrator: Q1 : Is it even possible to be addicted to coffee? Q2 : How can I energize myself? As humans, to properly distill the narrator's intention from the events and situations they describe, we need to apply a certain degree of social intuition (Conzelmann, 2012; Conzelmann et al., 2013; Baumgarten et al., 2015; Kehler and Rohde, 2017). As an example, consider the goals of a narrator sharing the personal story in Figure 1. We are presented with a wealth of information about the narrator's general sleep patterns, about a particular sleep deprivation situation and its physiological effects, about an upcoming lesson, about coffee intake, its effects, and potential health impacts, and about the current location of the narrator and its impact on food supply. Taking these facts separately, we can imagine providing advice on how to get more sleep, on whether to postpone the lesson, Task Desired output A Question generation What do I need to do in 2 hours? on how to get food delivered, or on the risks of caffeine intake. However, given how the narrative is constructed, we can intuit that the more likely goal of the narrator is to get advice on how to overcome the effects of sleep deprivation so that they can be alert for the upcoming programming lesson. Importantly, the primary goal of our proposed task is not to understand details about the narrator's actions in the story (Why is the narrator tired?, When do they need to go to the lesson?), but to infer the reason why the narrator is sharing this story (i.e., To get advice on how to stay alert in the next few hours.). That is, we are not concerned with the intradiegetic aspects of the narrative, but with the extradiegetic intention of the narrator in sharing the story. While an understanding of the former is likely necessary for the latter, it is often not sufficient. In this work, we introduce a task and a large dataset to evaluate the capabilities of automated systems to infer the narrator's (extradiegetic) intention in constructing and sharing an advice-seeking personal story. This complements existing narrative understanding tasks which focus on testing semantic understanding of events, actors and their (intradiegetic) intentions within the narrative itself. Table 1 contrasts the goals of these existing narrative understanding tasks with that of inferring a narrator's advice-seeking intention, in the context of our introductory example. Formally, we implement the task as a binary choice cloze test, where the goal is to identify which of two candidate advice-seeking questions was actually asked by the narrator of a given personal narrative. Beyond collecting a large and diverse set of realistic personal stories that contain an advice-seeking question, the main challenge in constructing this task is finding a plausible alternative advice-seeking question for each given narrative. To address this challenge, we develop a methodology for identifying such questions by exploiting both the commonalities in experiences people share online and the diversity of possible advice-seeking intentions that can be tied to similar experiences. By applying our methodology to a large collection of online personal narratives, we construct a dataset of over 20,000 cloze test instances, covering a very broad spectrum of realistic advice-seeking situations. 1 Each instance contains a narrative that is matched with two advice-seeking questions, one of which is actually asked by the narrator (Q2 in our introductory example), and the other semantically related to the narrative (Q1). We use human annotations to judge the relative difficulty of different subsets of the test instances and the type of reasoning necessary to solve them. We find that more than half of the instances contain pairs of questions that are not only semantically related to the narratives but also do not contain any explicit factual mismatches with the stories. These are thus unsolvable by pure logical reasoning and require some degree of common sense or social intuition. And indeed, simple 1 The dataset is available at https://github.com/ CornellNLP/ASQ . baseline approaches perform worse on these types of instances, highlighting the need for more direct modeling of the intention of the narrator. To summarize, in this work we:  formulate the task of inferring advice-seeking intents from personal narratives (Section 2);  develop a methodology to construct a large dataset of personal narratives matched with plausible options for a dvices eeking q uestions (the ASQ dataset) to be used for this task (Section 3);  show the task is viable and evaluate the relative difficulty of its items (Sections 4 & 5). We end by discussing the practical implications of endowing systems with the capability to infer advice-seeking intentions and use our results to identify avenues for developing better models. To evaluate the capability of automated systems to infer advice-seeking intentions, we formulate a cloze-style binary choice test where the system is presented with a personal narrative and is required to choose between two plausible candidate questions: one actually asked by the narrator and the other one not (as exemplified in Figure 1). We motivate the task by contrasting it with other (narrative) understanding tasks (Section 2.1), and provide the rationale for this particular formulation by discussing its advantages (Section 2.2). There are many tasks involving reading comprehension in general, and story understanding in particular. Given a narrative, there are a few broad categories of questions that may be asked to test different types and degrees of understanding. Table 1 follows directly from the discussion below, by contrasting the goal of our task with those of (intradiegetic) narrative understanding tasks in the context of our introductory example. A: What happened in the story? The most direct approach to test story understanding is to check whether the reader could comprehend the events and actions that occur within the story. This requires semantic understanding, but nothing more. This type of task can be set up in various forms, as the system can be asked to summarize the story (summarization, see Nenkova (2011); Allah-yari et al. (2017) for surveys), generate a question that is answerable from the text (question generation (Du et al., 2017)), or answer a question for which the information can be retrieved or reasoned directly from the story (reading comprehension, see Chen (2018) for a survey; notable datasets include MCTest (Richardson, 2013) and NarrativeQA (Kocisky et al., 2018)). B: What might happen next? While reading the story, people not only grasp and process the events that already occurred but also have some intuition of its likely trajectory. Related tasks include the narrative cloze task (Chambers and Ju-rafsky, 2008), the story cloze test (Mostafazadeh et al., 2016; Chaturvedi et al., 2016), and its generative versions (Guan et al., 2019). These tasks might require some common sense reasoning on top of semantic understanding; the fact that they aim to predict the future might require a deeper level of understanding than the previous tasks. C: What can we infer about the characters? When people read a narrative, they not only grasp the facts explicitly stated in the story, but also make inferences about the actors' mental states, such as their attitudes and desires, as the story unfolds. 2 Oftentimes, such an understanding requires inference, either logical or based on common sense reasoning. Such tasks can aim to generate the likely intents and reactions from the actors involved in the events (Rashkin et al., 2018a,b), or to determine whether a given desire of the protagonist was fulfilled (Rahimtoroghi et al., 2017). D: What is the intention of the narrator in sharing their story? While these prior tasks cover a wide range of angles to narrative understanding, they take an intradiegetic view by focusing on understanding the story itself. We propose another dimension to this line of work by taking an outside-the-story (extradiegetic) perspective 3 and aiming to understand why the story is shared by the narrator, potentially inferred from how the narrator decides to construct it. In particular, the task introduced here is to infer the advice-seeking intention of the narrator. 4 2 See Bratman (1987) for an account of the Belief-Desire-Intention model of human practical reasoning. 3 Recognizing the importance of these two different perspectives for story understanding, Swanson et al. (2017) attempted to classify narrative clauses into intradiegetic vs. extradiegetic levels. 4 Sharing personal stories can have other goals, e.g., therapeutic (Pennebaker, 1997; Pennebaker and Seagal, 1999). We argue that solving this task requires not only the semantic understanding and common sense reasoning involved in prior tasks but also a certain degree of social intuition. To uncover the goals of the narrator, one needs to find cues in the narrative constructionwhat has been selectively included or emphasized, and what might have been purposefully omitted (Labov, 1972). In fact, such intention-understanding tasks are often included in social intelligence tests (Conzelmann et al., 2013; Baumgarten et al., 2015). To evaluate the capacity of NLP systems to solve this task, we consider a binary choice cloze test formulation for two main reasons. First, it allows natural ground-truth labels: often, when people share their personal experiences to seek advice, they add explicit requests for the information they are seeking. After removing these requests from the narratives, we can use them as proxies for the narrators' intentions. Second, the binary choice operationalization also has the advantage of nonambiguity in evaluations and ease of comparisons between systems (as opposed to a generation task). It is worth noting that our dataset is constructed in a way that allows easy modifications into other task formats if so desired. For instance, the methodology of identifying a plausible false choice for a given narrative could be applied multiple times to extend the task to a more difficult multiple-choice version. Similarly, by ignoring the incorrect question in each instance, our dataset can be used as a source for a new generation task, i.e., generating the advice-seeking question from the given narrative. For a meaningful implementation of the proposed task, the collection of test instances must conform to several expectations, in terms of both the narratives and their (actual) advice-seeking questions. In what follows we outline these desiderata and our method for collecting instances that meet them (Section 3.1). Furthermore, as with any multiple-choice cloze test formulation, the difficulty of each test instance largely depends on how plausible the alternative answers are. Yet, finding plausible (but not actually correct) alternatives automatically is challenging. Not surprisingly, many of the cloze-style multiple-choice datasets use humans to write these alternatives (Mostafazadeh et al., 2016; Xie et al., 2018), limiting their scalability. We tackle this challenge by developing a methodology that exploits both the commonalities in human experiences shared online and the diversity in the types of advice needed for similar situations under different circumstances (Section 3.2). Narratives desiderata. As a pre-requisite, we need to start from personal narratives containing advice-seeking needs that are explicitly expressed (as questions), and that can be removed to form the cloze test instances. 5 Ideally, these narratives would cover a broad range of topics, in order to be able to test how well a system can generalize to a diverse range of real-life scenarios, rather than apply only to restricted and artificial settings. Question desiderata. Not all questions contained within an advice-seeking narrative are suitable for our task. Some of the questions might be too general, while others might be rhetorical. For instance, Any advice? holds no particular connection with the context of the narrative in which it appears. To contribute to meaningful test instances, questions need to meet a level of relevance and specificity such that (at least) humans could match them with the narratives from which they are extracted. Data source. We start from a dataset of over 415,000 advice-seeking posts collected from the subreddit r/Advice, which self-defines as a place where anyone can seek advice on any subject. 6 We only use publicly available data and will honor the authors' rights to remove their posts. Applying cloze. For each post, we strip off all questions that appear in any position of the post, including the post title. 7 We keep the remaining narratives as the cloze texts. 8 Figure 2 shows how the cloze transforma-5 An interesting future work avenue could be considering narratives that only have implicit advice-seeking intentions. 6 We start from an existing collection of Reddit posts (Tan and Lee, 2015) which we supplement with The Baumgartner Reddit Corpus retrieved via Pushshift API on Nov. 21, 2018. 7 To identify questions, we use the simple heuristic of looking for sentences that end with ?' or start with why, how, am, is, are, do, does, did, can, could, should, would . 8 To ensure that the cloze text can provide sufficient context, yet are not overly verbose, we only consider cloze texts that are 50-300 tokens long. This is a choice we made prior to any experiments, and we do not claim it is the optimal range to set up the task. Selecting ground-truth test answers. 9 We select candidate ground-truth answers for the cloze test as the ?-ending sentences removed from narratives. In order to keep only well-formed information-seeking questions, we filter the candidate questions by keeping only those that start with interrogatives 10 or any, anyone, help, advice, thoughts . To further discard questions that are too general, we compute a simple specificity score S ( q ) of a question q containing the set of words 9 As it happens, test answers are actually questions. 10 We consider the following set of words as interrogatives: what, when, why, where, which, who, whom, whose, how, am, is, are, was, were, do, does, did, has, have, had, can, could, shall, should, will, would, may, might, must . { w 1 , w 2 , . . . , w N } as its maximum inverse document frequency (idf): S ( q ) = S ( { w 1 , w 2 , . . . , w N } ) = max i  N idf ( w i ) , and filter out questions for which S ( q ) < 5 or questions that have less than 5 words. At the end of this selection process, from the example post in Figure 2, Help? and What has worked for you? are discarded and the title question is kept as the ground-truth answer to this cloze instance. If multiple questions survive the filtering process, we select one at random. Diversity evaluation. To verify that the resulting data has broad topical diversity in both narratives and questions, we perform a two-step clustering analysis. First, we use singular value decomposition on tf-idf transformed narratives to obtain their vector representations, we then cluster similar narratives using k-means to surface underlying topics. Next, for each topic, we extract nouns and verbs from the questions attached to each narrative in the topic, and surface common question keywords as those with high document frequency within the topic, correcting for their global document frequency (via subtraction). To provide a qualitative feel of the diversity of the data, Table 2 shows a selection of the resulting narrative topics and question keywords, together with example questions (corresponding narratives can be found in the data release). We find a wide range of experiences represented in the narratives, from relationships to student life to apartment rentals. Furthermore, within each narrative topic, there is a variety of question types; for instance, questions related to housing could be about dealing with roommates, paying rent, or choosing a city to live in. To find plausible alternative answer options for each candidate cloze test instance, one direct approach could be to find questions that are semantically related to the ground-truth question. However, there are two underlying problems with this approach. First, the task of finding semantically similar questions is itself very challenging (Haponchyk et al., 2018), given their terseness and lack of context. Second, semantic similarity is arguably a different concept from plausibility with respect to a narrative. For example, the two questions in the introductory example are semantically distant, but they are both plausible in the context of the narrative. Our main intuition in solving this problem is that individuals who are in similar situations tend to have advice-seeking intentions that are related. For each candidate cloze test narrative instance, we can thus search for a similar narrative first (by exploiting commonalities in experiences people share online) and then select an advice-seeking question from that narrative as the alternative answer for the test. Narrative pairing. To operationalize this intuition, we first find pairs of similar narratives based on the cosine similarity of their tf-idf representations. 11 A greedy search based on this similarity metric results in a set of pairs of related narratives (N 1 , N 2 ) with their respective advice-seeking questions (qn 1 , qn 2 ) identified in the previous step. Narrative masking. At this point, the pair of advice-seeking questions could be used with either narrative to form a test instance. For example, Figure 3 shows the other possible cloze instance corresponding to the introductory example if we were to use the other narrative in the narrative pair. This, however, would arguably be a poor 11 We consider both unigrams and bigrams, and set a minimum document frequency of 50. We also remove likely duplicates (cosine > 0 . 8 ) and cases for which the similarity between narratives is too low (cosine < 0 . 1 ). We have also experimented with embedding-based representations to compute cosine similarities from, but they do not seem to produce qualitatively better pairings upon inspection. Masked narrative : I've noticed something, over the past few years I've gained a habit of drinking coffee. The average day is about six cups, but it can exceed that sometimes (8 or so). The only reason I question my habit is cause I'm up at 4AM right now cause I couldn't fall asleep. I honestly have a headache in the morning until I drink a cup of coffee. I'll have some for essentially no reason, I'll just make some out of a urge almost. Q1 : Is it even possible to be addicted to coffee? Q2 : How can I energize myself? test instance since Q2 is hardly applicable to this other narrative. More generally, we want to ensure that our choice of which narrative (N i ) to include in the cloze test optimizes the plausibility of the question pair (qn 1 , qn 2 ). To achieve this, we compute the similarity between each narrative in the pair and each of the two respective questions, 12 and select the narrative that maximizes the minimum question-narrative similarity. Formally, N i = arg max i MIN { sim ( N i , qn 1 ) , sim ( N i , qn 2 ) } . Importantly, this selection criterion is purposely symmetric with respect to the two questions in order to avoid introducing any unnatural preference between the two that a classifier (with no access to the masked narrative) could exploit. As a final check, we ensure that in each cloze instance the two questions are neither too similar to each other (and thus indistinguishable) nor too dissimilar (which may indicate unsatisfactory narrative pairings). To this end, we discard instances in which the questions have extremely high or low surface similarity according to their InferSent (Conneau et al., 2017) sentence embeddings. 13 This process leaves us with a total of 21,865 instances. A detailed account of the number of instances filtered at different stages of the construction process can be found in the Appendix. 12 To account for the terseness of the questions, we represent both narratives and questions with tf-idf weighted GloVe embeddings (Pennington et al., 2014) and compute the cosine similarity between them. 13 We set a lower bound of 0.8 and an upper bound of 0.95. We choose this representation because questions are short and thus we anticipate tf-idf representation to be less informative. To understand the feasibility of the task, as well as the relative difficulty of the items in the dataset, eight non-author annotators labeled a random sample of 200 instances. 14 Each annotator is asked to choose first, out of the two candidate questions, which they consider to be more likely to have been asked by the narrator. Overall, human annotators achieve an accuracy of 90% (Cohen's  = 0.79), 15 showing that humans can indeed recover the advice-seeking intentions of the narrators, and thus validating the feasibility of the task. 16 We are also interested in understanding the types of skills needed to solve the task. In particular, we want to estimate the proportion of the task instances that can not be solved by mere factual reasoning. To this end, we ask humans to identify candidate questions that contain a factual mismatch with the narrative, making them E xplicitly incompatible; 57% of the annotated instances do not contain any such mismatches in any of the questions. Similarly, we want to estimate how many instances require common sense expectations about the behavior of the protagonist (within the story). So we ask annotators to mark questions as being I mplicitly incompatible if they do not contain any factual mismatches, but they are incompatible with what can be inferred implicitly about events and characters in the story. The questions that are neither explicitly nor implicitly incompatible would be labeled as being C ompatible, and as either L ikely or U nlikely to represent the narrators' intentions. Test items in our data forcing a choice between C ompatible questions are expected to be the hardest to solve, as they might require a certain degree of social intuition in addition to factual and common sense reasoning. Table 3 provides an example narrative and one representative question from each of the above-mentioned categories. 17 Table 4 shows a human performance breakdown according to some of the most common types of instances in our data. 18 As expected, instances 14 See the Appendix for detailed annotation instructions. 15 We obtained a second round of annotations on a subset of 75 task instances to compute agreement statistics. 16 By construction, random accuracy is 50%. 17 The example is adapted from our instructions to annotators, which includes further explanations for these categories. See the Appendix for details. 18 See the Appendix for some representative examples for selected question pair types in our data. Narrative : I asked a girl that I really like if she would like to get coffee sometime. She said she's really busy but that we'll see. I can't get her off my mind and I spend all day waiting for her to tell me she's free. involving only compatible questions (C + C) are harder to solve, 19 as they might require some social intuition, whereas when explicit contradictions exist (C + E), they are perfectly solvable. We also note that humans can perfectly solve the subset of task instances (L + { U, I } ) that exhibit perceived qualitative differences between the actual and the alternative questions, but nevertheless, require more than semantic understanding (and sometimes require social intuition). 19 We also concede that some of the instances in this category may be unsolvable, e.g., when the wrong question fits the narrative just as well. We divide our data into a 8,865-2,500 train-test split and have reserved 10,000 instances as a held-out set. 20 In Table 5 we report accuracy for the best-performing model on the (never-before-seen) held-out for a simple similarity-based method and for a deep learning method. Narrative-question similarity. We expect that questions would show greater similarity to narratives they are removed from. We thus establish a narrative-question similarity baseline by considering features based on cosine similarities between narrative and questions, with text represented as tf-idf vectors, tf-idf weighted GloVe embeddings, averaged GloVe embeddings, as well as word overlap between content words, all combined in a logistic regression model. Finetuned transformer LM. We also use a Finetuned Transformer LM model (Radford et al., 2018), which was shown to perform competitively on a diverse set of NLP tasks, achieving state-of-the-art results on the story cloze test. 21 5.1 Error analysis Required skills. As shown in Table 4, systems perform worst on items that do not exhibit any (implicit or explicit) mismatches (C + C), and thus might require some social intuition. Importantly, the largest gap between baseline and human performance (25%) is on the subset of items that can not be solved based solely on a semantic understanding (L + { U, I } ). These results underline the need for models that can combine common sense reasoning about the events within the story with an intuition about the intention of the narrator. Question concreteness. Questions may also differ in how concrete they are. In a preliminary analysis aimed at understanding how this property affects performance, we compare words used 20 The set annotated by humans is disjoint. 21 We fine-tune with our training set on top of the pretrained transformer language model, using the implementation from https://github.com/huggingface/ pytorch-openai-transformer-lm . in ground-truth questions that the best-performing model predicts correctly with those used in questions that are classified incorrectly. We observe that questions that are predicted correctly have significantly higher average inverse document frequencies (t-test p < 0 . 01 ). Intuitively, these more specific questions may be more concrete in nature, making them easier to connect to the narratives to which they belong. We also find that some common interrogatives have skewed distributions. For instance, questions starting with Is are less likely to be classified correctly than those starting with How . A cursory manual investigation suggests that this can also be tied by concreteness, with the latter type of questions appearing to be more concrete than the former. One broad motivation behind our work is to eventually help better support personalized informational needs (Teevan et al., 2007). This connects to several related lines of work that were not previously discussed. Query/question intents. Datasets and models are proposed for understanding user intents behind search queries (Radlinski et al., 2010; Fariha et al., 2018), or even more generally, user questions (Haponchyk et al., 2018). To complement this line of work that looks at user intents behind the explicit request, our task aims to uncover user intents when they are implied in personal narratives (without access to the explicit question). Conversational search/QA. One way to better satisfy user intents is by making such processes collaborative (Morris and Horvitz, 2007; Morris, 2013), or conversational (Radlinski and Craswell, 2017). Conversational QA datasets (Choi et al., 2018; Reddy et al., 2019) have been introduced to help develop systems with such capability. Social QA. Some questions posed by users are inherently more social in nature, and require more nuanced contextual understanding (Harabagiu, 2008). The social nature may affect how people ask questions (Dahiya and Talukdar, 2016; Rao and Daume III, 2018), and pose challenges for identifying appropriate answers (Shtok et al., 2012; Zhang et al., 2017). In this work, we introduce the new task of inferring advice-seeking intentions from personal narratives, a methodology for creating appropriate test instances for this task and the ASQ dataset. This task complements existing (intradiegetic) narrative understanding tasks by focusing on extradiegetic aspects of the narrative: in order to understand Why is the narrator sharing this?, we often need to apply a certain degree of common sense and social intuition. From a practical perspective, this extradiegetic capability is a prerequisite to properly address personalized information needs that are constrained by personal circumstances described as free-form personal stories. Currently, to address these types of information needs, people seek (or even hire) other individuals with relevant experience or expertise. As with conversational search (Radlinski and Craswell, 2017), we can envision systems that can more directly address complex information needs by better understanding the circumstances and intentions of the user. Our analysis of the human and baseline performance on different types of test instances points to interesting avenues for future work, both in terms of designing better-performing systems and in terms of constructing better test data. We envision that (intradiegetic) narrative understanding could help identify the components of the narrative that are most relevant to the advice-seeking goal. For example, identifying the narrator's intentions and desires within the story (Rashkin et al., 2018b), and whether these desires are fulfilled (Rahimtoroghi et al., 2017) could help focus the attention of the model, especially when dealing with less concrete questions. Furthermore, a better representation of the structure of the narrative (Ouyang and McKeown, 2014), in terms of discourse acts (Elson, 2012) and sentiment flow (Ouyang and McKeown, 2015), could also help distinguish between spurious and essential circumstances of the narratives. In terms of improving the task itself and the methodology for creating testing instances that better approximate the inferential task, we note a few possible directions. Firstly, better narrative modeling could lead to higher quality matching. Similarly, better representation of the questions can help select more appropriate candidate options (e.g., currently 6% of the questions are deemed by the annotators to be too general). In addition, the generative version of the task, when appropriately evaluated, could be a closer operationalization for intention inference, and also offer more potential for practical uses. Finally, future work could expand on our methodology to formulate other more general tasks aiming to understand the reasons why a person is sharing a personal story. While we have focused on narratives shared with the intention of seeking advice, people may also share stories to express emotions, to entertain or educate others. A better understanding of these different (explicit or implicit) intentions could lead to more personalized and empathetic human-computer interaction. Acknowledgments. The authors thank Tom Davidson, Tsung-Yu Hou, Qian Huang, Hajin Lim, Laure Thompson, Andrew Wang, Xiaozhi Wang and Justine Zhang for helping with the annotations. We are grateful to Thorsten Joachims, Avery Quinn Smith and Todd Cullen for helping us when our server crashed on the day of the deadline while testing the model on the held-out set, to Lillian Lee, Andrew Wang, Justine Zhang and the anonymous reviewers for their helpful comments, and to Fernando Pereira for the early discussions that inspired this research direction. This work is supported in part by NSF CAREER award IIS-1750615 and NSF Grant SES-1741441.
abstain abstain objective objective abstain method abstain abstain method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method abstain objective objective abstain abstain method abstain objective objective abstain objective method abstain objective result abstain other abstain other objective method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain method method method abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain result objective method abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain other other other objective other other other other other other abstain method abstain abstain abstain method abstain abstain abstain result abstain abstain abstain abstain method abstain abstain abstain abstain abstain