Long-form answers, consisting of multiple sentences, can provide nuanced and comprehensive answers to a broader set of questions. To better understand this complex and understudied task, we study the functional structure of long-form answers collected from three datasets, ELI5 (Fan et al., 2019), WebGPT (Nakano et al., 2021) and Natural Questions (Kwiatkowski et al., 2019). Our main goal is to understand how humans organize information to craft complex answers. We develop an ontology of six sentence-level functional roles for long-form answers, and annotate 3.9k sentences in 640 answer paragraphs. Different answer collection methods manifest in different discourse structures. We further analyze model-generated answers  finding that annotators agree less with each other when annotating model-generated answers compared to annotating human-written answers. Our annotated data enables training a strong classifier that can be used for automatic analysis. We hope our work can inspire future research on discourse-level modeling and evaluation of long-form QA systems. 1 1 Introduction While many information seeking questions can be answered by a short text span, requiring a short span answer significantly limits the types of questions that can be addressed as well as the extent of information that can be conveyed. Recent work (Fan et al., 2019; Krishna et al., 2021; Nakano et al., 2021) explored long-form answers, where answers are free-form texts consisting of multiple sentences. Such long-form answers provide flexible space where the answerer can provide a nuanced answer, incorporating their confidence and sources of their knowledge. Thus the answer sentences form a discourse where the answerers provide information, hedge, explain, provide examples, point 1 Our data, code and datasheet are available at https: //github.com/utcsnlp/lfqa_discourse . be structured and organized coherently. We take a linguistically informed approach to understand the structure of long-form answers, designing six communicative functions of sentences in long-form answers (which we call roles ). 2 Our framework combines functional structures with the notion of information salience by designating a role for sentences that convey the main message of an answer. Other roles include signaling the organization of the answer, directly answering the question, giving an example, providing background information, and so on. About a half of the sentences in long-form answers we study serve roles other than providing an answer to the question. We collect discourse annotations on three long-form question answering (LFQA) datasets, ELI5 (Fan et al., 2019), WebGPT (Nakano et al., 2021) and Natural Questions (NQ) (Kwiatkowski et al., 2019). Figure 1 contains an example annotation on each dataset. While all three contain paragraph-length answers needed for complex queries, they are collected in distinct manners  answers in ELI5 are written by Reddit users; answers in WebGPT are written by annotators who searched documents on a web interface and heavily quoted those documents to form an answer, and answers in NQ are pre-existing paragraphs from Wikipedia corpus. We collect three-way annotations for 3.9k sentences (  700 question-answer pairs across three datasets). We also annotate a small number of model-generated answers from a recent long-form question answering (LFQA) system (Krishna et al., 2021) and provide rich analysis of their discourse structure. In all three datasets, we observe appearance of most proposed functional roles, but with different proportions. Answers in ELI5 contains more examples and elaborations, while answers extracted 2 Functional structures have been studied in various other domains (discussed in Sections 2 and 7). worked on a window-washing robot that cleaned acres of rooftops over a huge commercial greenhouse. Worked great, except when it didn't, and would either break down completely or just get lost and start climbing the wrong parts of the structure. Then repair techs and manual window washers still have to be employed. I think this ends up being a cost/benefit problem where the reliability of our robots and price of implementation isn't quite at the point where it makes this commercially viable for skyscrapers. For what it's worth, I think the Twin Towers actually used a washer robot on the upper floors to limited success. A subject matter expert ( SME ) or domain expert is a person who is an authority in a particular area or topic. The development of accounting software requires knowledge in two di  erent domains : accounting and software. Some of the development workers may be experts in one domain and not the other. A SME should also have basic knowledge of other technical subjects . Analyzing a subset of ELI5 and WebGPT, we also identify a big gap in lexical overlap between long-form answer and evidence passages across all functional roles. Lastly, we found that human agreement of the discourse roles of model-generated answers are much lower than human-written ones, reflecting the difficulty for humans to process model-generated answers. With the data collected, we present a competitive role classifier, which performs on par with human when trained with our annotated data and can be used for automatic discourse analysis. We further envision using functional roles for controllable long-form generations, concise answer generation, and improved evaluation metrics for LFQA. We study the discourse structure of long-form answers based on functional roles of sentences in the paragraph. Functional structures characterize the communicative role a linguistic unit plays; as such, they vary across genres as the goals of communication also vary. In scientific or technical articles, these roles can be background, method, findings (Kircz, 1991; Liddy, 1991; Mizuta et al., 2006), while in news, they can be main event or anecdotes (Van Dijk, 2013; Choubey et al., 2020). These structures are related to, though distinct from, coherence discourse structures (Hobbs, 1985). The latter characterizes how each unit (e.g., adjacent clauses or sentences) relates to others through semantic relations such as temporal, causal, etc.; such structures can be trees that hierarchically relate adjacent units (Mann and Thompson, 1988) or graphs (Lascarides and Asher, 2008). In contrast, functional roles describe how information is organized to serve the communication goal, in our case, providing the answer. We developed our ontology by examining long-form answers in online community forums (subred-dit Explain Like I'm Five (ELI5)) and Wikipedia passages, hence answers derived from different domains (e.g., textbooks) can contain roles beyond our ontology. We describe our six sentence-level discourse roles for long-form answers here: Answer-Summary (Sum), Answer (Ans). An answer sentence directly addresses the question. Here we distinguish between the the main content of the answer (henceforth answer summary ) vs. sentences which explain or elaborate on the summary. The summaries play a more salient role than non-summary answer sentences, and can often suffice by themselves as the answer to the question. This is akin to argumentation structure that hierarchically arranges main claims and supporting 3557 arguments (Peldszus and Stede, 2013), and news structure that differentiates between main vs. supporting events (Van Dijk, 2013). Organizational sentences (Org.) Rather than conveying information of the answer, the major role of an organizational sentence is to inform the reader how the answer will be structured. We found two main types of such sentences; the first signals an upcoming set of items of parallel importance: [A] : There are a few reasons candidates with no chance" to win keep running . 1) They enjoy cam-paigning[...] The other type indicates that part of the answer is upcoming amidst an established flow; in the example below, the answerer used a hypophora: [A] : It might actually be a mosquito bite. I find the odd mosquito in my house in the winter from time to time, and I'm in Canada.[...] So why does it happen more often when you shower? It's largely because [...] Examples (Ex.) Often people provide examples in answers; these are linguistically distinct from other answer sentences in the sense that they are more specific towards a particular entity, concept, or situation. This pattern of language specificity can also be found in example-related discourse relations (Louis and Nenkova, 2011; Li and Nenkova, 2015), or through entity instantiation (MacKinlay and Markert, 2011): [Q] : What is it about electricity that kills you? [A] : [...] For example, static electricity consists of tens of thousands of volts, but basically no amps. [...] We found that examples in human answers are often not signaled explicitly, and often contain hypothetical situations: [Q] : Were major news outlets established with political bias or was it formed over time? [A] : [...]This is impossible due to the problem of anchoring. Consider a world where people on the right want the tax rate to be 1% lower and people on the left want the tax rate to be 1% higher [...] Auxiliary information (Aux.) These sentences provide information that are related to what is discussed in the answer, but not asked in the question. It could be background knowledge that the answerer deemed necessary or helpful, e.g., [Q] : Why is it better to use cloning software instead of just copying and pasting the entire drive? [A] : When you install an operating system, it sets up what's called a master file table, which [...] are important for the OS to work properly. [...] Simply copy-pasting files doesn't copy either of these, meaning if you want to back up an OS installation you should clone the disk instead. or related content that extends the question, e.g., [Q] : what is the difference between mandi and kabsa? [A] : [...] A popular way of preparing meat is called mandi. [...] Another way of preparing and serving meat for kabsa is mathbi , where seasoned meat is grilled on flat stones that are placed on top of burning embers. Notably, the removal of auxiliary information would still leave the answer itself intact. Miscellaneous (Misc.) We observe various roles that, although less frequent, show up consistently in human answers. We group them into a miscellaneous role and list them below. (a) Some sentences specify the limitation of the answer by narrowing down the scope of the answer to an open-ended question. (b) Some sentences state where the answer came from and thus put the answer into context. [Q] : Why Does a thermostat require the user to switch between heat and cool modes, as opposed to just setting the desired temperature? [A] : The person who installed my heat pump (which has all three modes) explained this to me. [...] (c) Some sentences point to other resources that might contain the answers. (d) Answerers also express sentiment towards other responses or the question itself. [Q] : Why did Catholicism embrace celibacy and why did Protestantism reject it? [A] : Good God, the amount of misinformation upvoted is hurting. [...] [Q] : Could you Explain Schrdinger's Cat to me LI5? [A] : [...] It's a pretty cool thought experiment, but it doesn't mean too much in our everyday lives. As our ontology does not provide an exhaustive list of the functional roles, we instructed our annotators to annotate other roles not covered by our ontology as Miscellaneous as well. We randomly sample examples from three LFQA datasets and filter answers with more than 15 sentences and those with less than 3 sentences. 3 We briefly describe each dataset below. 4 ELI5 / ELI5-model ELI5 consists of QA pairs where the questions and answers are retrieved from the subreddit r/explainlikeimfive . The answers in ELI5 are of varying quality and style. While the original dataset consists of (question, answer) pairs, recent benchmark (Petroni et al., 2021) annotated a subset of examples with relevant Wikipedia paragraphs, which we used for analysis in Section 4. In addition to answers in the original datasets, we annotate a small number of model-generated answers from Krishna et al. (2021) (we refer this set as ELI5-model), a state-of-the art LFQA system on ELI5. WebGPT Nakano et al. (2021) presented a new LFQA dataset and model; with the goal of building a model that can search and navigate the web to compose a long-form answer. While they reuse questions from ELI5, they newly collect answers from trained human annotators who were instructed to first search for related documents using a search engine and then construct the answers with reference to those documents. The collected data (denoted as human demonstration consisting of question, answer, a set of evidence documents, and mapping from the answer to the evidence document) are used to finetune GPT-3 (Brown et al., 2020) to generate long-form answers. Natural Questions (NQ) NQ contains questions from Google search queries, which is paired with a relevant Wikipedia article and an answer in the article if the article answers the question. They annotate paragraph-level answer as well as short span answer inside the paragraph answer if it exists. In open retrieval QA, researchers (Lee et al., 2019) filtered questions with paragraph level answers for its 3 We used Stanza (Qi et al., 2020) to split long-form answers into sentences. This process removes 42%, 28% and 34% from ELI5, WebGPT and NQ respectively. 4 Our data is sourced from the validation split of ELI5 from the KILT (Petroni et al., 2021) benchmark, the testing portion from WebGPT (their samples are publicly hosted at https://openaipublic.blob.core.windows. net/webgpt-answer-viewer/index.html , which answers questions from the ELI5 test set), and the validation split from Natural Questions. difficulty of evaluation and only look at questions with short span answer. We create a filtered set of NQ that focuses on paragraph-level answers containing complex queries. 5 While many NQ questions can be answered with a short entity (e.g., how many episodes in season 2 breaking bad? ), many others questions require paragraph length answer (e.g., what does the word china mean in chinese? ). This provides a complementary view compared to the other two datasets, as the answers are not written specifically for the questions but harvested from pre-written Wikipedia paragraphs. Thus, this simulates scenarios where model retrieves paragraphs instead of generating them. We have a two-stage annotation process: annotators first determine the validity of the QA pair, and proceed to discourse annotation only if they consider the QA pair valid. We define the QA pair as valid if (1) the question is interpretable, (2) the question does not have presuppositions rejected by the answer, (3) the question does not contain more than one sub-question, and (4) the proposed answer properly addresses the question. Examples of the invalid QA pair identified are in A.1. 6 We collect the first stage annotation from US-based crowdsource workers on Amazon Mechanical Turk and second stage annotation from undergraduate students majoring in linguistics, who are native speakers in English. 7 A total of 29 crowd-worker participated in our task, and six undergraduates annotated roles for a subset of QA pairs annotated as valid by crowdworkers. We first qualified and then provided training materials to both groups of annotators. The annotation guideline and interface can be found in A.4. We paid crowd workers $0.5 per example, and our undergraduate annotators $13 / hour. More details of data collection can be found in our datasheet. Table 1 presents the statistics of our annotated data. We collected validity annotations for 1.5K exam-5 Implementation details are in A.3. questions in our github repository. 6 The categories are not mutually exclusive, and we let annotators to pick any of them when an example belongs to multiple categories. ples and role annotations for about half of them. As our tasks are complex and somewhat subjective, we collected three way annotations. We consider a QA pair valid if all annotated it as valid, and invalid if more than two annotated it as invalid. If two annotators considered valid, we collect one additional annotation and consider it valid if and only if the additional annotator marked it as valid. 8 We consider the majority role (i.e. chosen by two or more than two annotators) as the gold label. When all annotators chose different roles, they resolved the disagreement through adjudication. We report inter-annotator agreement before the adjudication. Inter-annotator Agreement We find modest to high agreement for both annotation tasks: For crowdworkers, Fleiss Kappa was 0.51 for validity annotation. For student annotators, Fleiss Kappa was 0.44 for role annotation. Figure 2 shows the confusion matrix between pairs of annotations, with the numbers normalized by row and averaged across pairs of annotators. We observe frequent confusion between roles denoting different levels of information salience Answer vs. Answer-Summary, and Answer vs. Auxiliary Information, reflecting the nuance and subjectivity in judging what information is necessary to answer a complicated question. Examples can be found in A.2. 8 The Fleiss kappa for agreement improves to 0.70 after this re-annotation process. With our annotated data, we study the differences between the three types of long-form answers, namely answers provided by users in online community (ELI5), answers written by trained annotators through web search (WebGPT), and answers identified in Wikipedia passages (NQ). Q/A Validity Table 2 summarizes the portion of valid answers in the three datasets and the distribution of invalid reasons. NQ has the highest rate of invalid answer (15%). Upon manual inspection, we find that passages from Wikipedia written independently of the question often only partially address complex questions. This demonstrates the limitation of a fully extractive approach. Around 10% of the answers from ELI5 reject presupposition in the question, which is a common phenomena in information-seeking questions (Kim et al., 2021). WebGPT boasts the lowest invalid rate, showing the high quality of their collected answers. Role Distribution We study the distribution of roles in three datasets (Table 3). NQ shows the highest proportion of auxiliary information, as the paragraphs are written independent of the questions. In contrast, ELI5 contains more answer sentences and examples which provide explanation. Both ELI5 and WebGPT contain organizational sentences, demonstrating that it is commonly used when answerers assemble answers that cover more than one aspects. In all datasets, around half of the sentences serve roles other than directly answering the questions, such as providing auxiliary information or giving an example, which reflects the wide spectrum of information presented in a long-form answer. Relatively few sentences (less than 10%) are marked as miscellaneous, showing a high coverage of our ontology in the three LFQA datasets we investigated. Compared to ELI5, both WebGPT and NQ answers contain very little miscellaneous 3560 Data # of Annotated Sentences Role Answer Summary Auxiliary Example Org Misc ELI5 2670 30% 28% 18% 13% 1% 10% WebGPT 551 28% 35% 26% 8% 3% 0% NQ 695 21% 35% 39% 5% 0.4% 0.1% Total 3916 28% 30% 11% 23% 1% 7% Table 3: Sentence-level role distribution. sentences. This is partially because both datasets are more extractive and less personal, without sentences which serve the role of various kinds of communication from answerers to question askers (e.g. expressing sentiments, pointing to other resources) that are commonly seen in online community fo-rum. Discourse Structure Figure 3 presents the distribution of each role per its relative location in the answer. Despite the significant differences in the proportion of different discourse roles, the positioning of the roles is similar across the datasets. Answer summary and organizational sentences typically locate at the beginning of the paragraph, examples and answers often in the middle, with an increasing portion of auxiliary information towards the end. The sentences belonging to miscellaneous role frequently position at the beginning or the end of the paragraph, instead of intervening in the mid-dle. WebGPT contains a higher portion of auxiliary information locating at the beginning of the passage, followed by the answer summary sentences. Answer Extractiveness One important aspect for long-form answer is whether the answer can be attributed to an external evidence document. While answers from NQ are directly extracted from Wikipedia passages, both ELI5 and WebGPT are written specifically for the question. To help with verification, both datasets provide evidence documents paired with the answer, and yet there are design differences between the two. Answerer (an-notators) of WebGPT were instructed to answer the question based on the evidence documents returned by a search engine, while answers from ELI5 were written first independently and later paired with relevant Wikipedia passages (Petroni et al., 2021). We found that such difference leads to different level of extractiveness of the answer, by calculating sentence-level lexical overlap (after removing stopwords) with the evidence document. Overall, WebGPT answers exhibit more lexical overlap (un-igram: 0.64, bigram: 0.36) with evidence document than ELI5 answers (unigram: 0.09, bigram: 0.01). Answer sentences with different roles also exhibit different levels of extractiveness (detailed role-level overlap can be found in Table 8 in the appendix). For ELI5 answers, sentences belonging to answer and summary roles have the highest overlap while example, auxiliary information and miscellaneous sentences are less grounded to external sources. For WebGPT, organizational sentences are the least extractive among all the roles. Having analyzed discourse roles of human-written long-form answers, we investigate the discourse structure of model-generated answers. This will allow us to quantitatively study the difference in terms of discourse structure across gold and generated answers, which we hope will cast insights to the linguistic quality of system outputs. Systems We study model-generated answers from a state-of-the-art LFQA system (Krishna et al., 2021). 9 Their model uses passage retriever (Guu et al., 2020), and generates answers based on the retrieved passage with a routing transformer model (Roy et al., 2021). Q/A Validity We collect validity annotation on 193 model-generated answers, and 78 are considered invalid, significantly higher ratio than that of human written answers (Table 2). The Fleiss's kappa of QA pair validity is 0.26 (and 0.61 after collecting one more label), substantially lower than the agreement on human written answers (0.51, 0.70) while annotated by the same set of annotators. Detailed distribution of invalid reason annotated can be found in Table 9. Despite the low agreement, 60 of them are marked as No valid answer by at least two annotators. The flaw of automatic measures was also pointed out by prior work (Krishna et al., 2021), which compares ROUGE between human-written and model-generated answers. Our study reiterates that the generated answers received high ROUGE score without answering the question. Roles We proceed to collect sentence-level role annotations on 115 valid generated long-form answers following the same annotation setup in Section 3, and hence our annotators were not asked to evaluate the correctness or the quality of the answers (e.g. whether the generated example makes sense), focusing on the functional roles of sentences only. We found that the annotators disagree substantially more as compared to the human-written answers, with a Fleiss kappa of 0.31 (vs. 0.45 for human-written answers), suggesting that the discourse structure of model-generated answers are less clear , even to our trained annotators. The answer role distribution of model-generated answers is very different from that of the human written answers (Figure 4). The generated answers contain more sentences which provide auxiliary information, and fewer summary sentences. This suggests that model-generated answers contain a higher portion of information tangentially related to what is asked in the question. Model-generated answers also contain fewer example and miscella-9 We sampled from four different model configurations reported in their paper, i.e. combination of nucleus sampling threshold p={0.6, 0.9}, and generation conditioning on {pre-dicted, random} passages. The answers we annotated achieved a ROUGE-L of 23.19, higher than that of human-written answers on the same set of questions (21.28). Overall, our results suggest that machine-generated long form answers are different from human-written answers, and judging their discourse structure is nontrivial for human annotators, resulting in lower agreement. Recent study (Karpinska et al., 2021) also showed that expert annotators showed lower agreement and took longer time to evaluate the coherence of story generated from large-scale language model. We study how models can identify the discourse role for each sentence in long-form answer in a valid QA pair. 10 Such a model can be beneficial for large-scale automatic analysis. Task and Data Given a question q and its long-form answer consisting of sentences s 1 , s 2 ...s n , the goal is to assign each answer sentence s i one of the six roles defined in Section 2. As we have annotated more examples from ELI5 dataset (411 answer paragraphs compared to around 100 paragraphs in other three datasets (WebGPT, NQ and ELI5-model)), we randomly split the ELI5 long-form answers into train, validation and test sets with a 70%/15%/15% ratio, and train the model on the training portion. We use all other annotated datasets for evaluation only. For model-generated answers, we filtered 185 out of 1080 sentences where model-generated sentences do not have a majority role. This setup also allows us to study domain transfer of role classification model. Metrics We report accuracy with respect to the majority role label (or adjudicated one, if majority doesn't exist) ( Acc ), match on any label from three annotators ( Match ), F1 score for each role and their macro average score Macro-F1 . Lower bounds We present two simple baselines to provide lower bounds: (1) Majority: We predict the most frequent labels in the training data: Answer-Summary . (2) Summary-lead: We predict first two sentences as Answer-Summary , and the rest of the sentences as Answer . Classification Models This baseline classifies each sentence independently. We use the [CLS] token from RoBERTa-Large model (Liu et al., 2019) which encodes [ question <q> ans 1 ... <start> ans i <end> ...], where ans i is the i th sentence in the answer. The training batch size is set to 64, with the initial learning rate as 5 e  5 . We used AdamW optimizer and a linear learning rate schedule. We train the model for 10 epochs and report the result of the checkpoint with best validation accuracy, averaged across three random seeds. Seq2Seq Models We use two variations (base, large) of T5 model (Raffel et al., 2020), which take the concatenation of question and answer sentences, and output the roles for each sentence sequentially. This model predicts the roles of all sentences in the answer as a single sequence. The input sequence is [ question [1] ans 1 [2] ans 2 ...], where ans i denotes the i th sentence in the answer, and the target output sequence is set to [ [1] role 1 [2] role 2 [3] ...], where role i is the corresponding role for ans i (e.g. Answer for the Answer role). We limit the input/output to be 512/128 tokens. For evaluating the predicted roles, we parse the output string to identify the role predicted for each sentence. We used the batch size of 16, initial learning rate of 1 e  4 with AdamW optimizer and a linear learning rate schedule. We train the model for 30 epochs and report the result of the checkpoint with the best validation accuracy, averaged across three random seeds. Human performance We provide two approximations for human performance: upperbound ( u ) and lowerbound ( l ). (1) Human ( u ): We compare each individual annotator's annotation with the majority label. This inflates human performance as one's own judgement affected the majority label. (2) Human ( l ): We compare all pairs of annotation and calculate average F1 and accuracy of all pairs. For Match , we compute the match for each annotation against the other two annotations. Table 4 reports the results on ELI5 test set. 11 All models outperform the majority and summary-lead baselines. The sequential prediction model 11 Results on validation set are in Table 11 in the appendix. (T5) significantly outperform classification model (RoBERTa) which makes a prediction per sentence. The roles with lower human agreement (auxiliary, organizational sentence, answer) also exhibit low model performances, reflecting the subjectivity and ambiguity of roles for some sentences. Overall, with a moderate amount of in-domain annotated data, our best model (T5-large) can reliably classify functional roles of sentences in the long-form answers, showing comparable performances to human lower bound. Table 5 reports the results on the three out-of-domain datasets, WebGPT, NQ and ELI5-model (model-generated answers). Human agreement numbers are comparable across all datasets (0.53-0.59 for lower bound, 0.73-0.78 for upper bound). While T5-large still exhibits the best overall performance, all learned models perform worse, partially as the role distribution has changed. Despite trained on the ELI5 dataset, role classification model also perform worse on model-generated answers (ELI5-model), echoing our observation that human annotators find it challenging to process the discourse structure of model-generated answers. Our pilot showed that training with in-domain data improved the performances consistently, but the evaluation is on a small subset (after setting apart some for training), so we do not report it here. We anticipate that automatic role classification is feasible given moderate amount of annotation for all three human-written long-form answer datasets we study. Discourse structure. Our work is closely related to functional structures defined through content types explored in other domains; prior work has affirmed the usefulness of these structures in downstream NLP tasks. In news, Choubey et al. (2020) adopted Van Dijk (2013)'s content schema cataloging events (e.g., main event, anecdotal), which they showed to improve the performance of event coreference resolution. In scientific writing, content types (e.g., background, methodology) are shown to be useful for summarization (Teufel and Moens, 2002; Cohan et al., 2018), information extraction (Mizuta et al., 2006; Liakata et al., 2012), and information retrieval (Kircz, 1991; Liddy, 1991). The discourse structure of argumentative texts (e.g., support, rebuttal) (Peldszus and Stede, 2013; Becker et al., 2016; Stab and Gurevych, 2017) has also been applied on argumentation mining. To the best of our knowledge, no prior work has studied the discourse structure of long-form answers. Question Answering. Recent work (Cao and Wang, 2021) have investigated the ontology of questions, which includes comparison questions, verification questions, judgement questions, etc. We construct the ontology of functional roles of answer sentences. One of the roles in our ontology is summary, yielding an extractive summarization dataset. This shares motivation with a line of work studying query-focused summarization (Xu and Lapata, 2020). Concurrent to our work, Su et al. (2022) studies improving faithfulness of long-form answer through predicting and focusing on salient information in retrieved evidence document. Lastly, our work build up on three datasets containing long-form answers (Kwiatkowski et al., 2019; Fan et al., 2019; Nakano et al., 2021) and extends the analysis of long-form answers from earlier studies (Krishna et al., 2021). We present a linguistically motivated study of long-form answers. We find humans employ various strategies  introducing sentences laying out the structure of the answer, proposing hypothetical and real examples, and summarizing main points  to organize information. Our discourse analysis characterizes three types of long-form answers and reveals deficient discourse structures of model-generated answers. Discourse analysis can be fruitful direction for evaluating long-form answers. For instance, highlighting summary sentence(s) or sentence-level discourse role could be helpful for human evaluators to dissect long-form answers, whose length has been found to be challenging for human evaluation (Krishna et al., 2021). Trained role classifier can also evaluate the discourse structure of model-generated answers. Future work can explore using sentences belonging to the summary role to design evaluation metrics that focuses on the core parts of the answer (Nenkova and Passon-neau, 2004), for assessing the correctness of generated the answer. Exploring controllable generation, such as encouraging models to provide summaries or examples, would be another exciting avenue for future work. We annotate existing, publicly available long-form question answering datasets which might contain incorrect and outdated information and societal biases. We collected annotations through crowd-sourcing platform and also by recruiting undergraduate annotators at our educational institution. We paid a reasonable hourly wage ($13/hour) to annotators and documented our data collection process with datasheet (Gebru et al., 2021). We include studies on the extractiveness of long-form answers (how much content can be grounded to evidence document) through a coarse measure of lexical overlap. This is connected to faithfulness and reducing hallucination of QA system. Our study is limited to English sources, and we hope future work can address analysis in other languages. This work was partially supported by NSF grants IIS-1850153, IIS-2107524. We thank Kalpesh Krishna and Mohit Iyyer for sharing the model predictions and human evaluation results. We would like to thank Tanya Goyal, Jiacheng Xu, Mohit Iyyer, anonymous reviewers and meta reviewer for providing constructive feedback to improve the draft. Lastly, we thank Maanasa V Darisi, Meona Khetra-pal, Matthew Micyk, Misty Peng, Payton Wages, Sydney C Willett and crowd workers for their help with the complex data annotation.
abstain method objective objective abstain result method abstain abstain abstain abstain other abstain method method abstain method method abstain abstain method method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain abstain abstain abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain method method abstain abstain abstain abstain method abstain abstain abstain method abstain other abstain abstain abstain abstain method abstain abstain method abstain method abstain method abstain abstain method method method abstain method method abstain method method abstain abstain abstain abstain abstain result abstain abstain abstain result result abstain other objective other other other method other other method abstain other abstain abstain method objective result abstain abstain abstain abstain abstain abstain method abstain abstain abstain method other other other other