Humanities scholars commonly provide evidence for claims that they make about a work of literature (e.g., a novel) in the form of quotations from the work. We collect a large-scale dataset (RELiC) of 78K literary quotations and surrounding critical analysis and use it to formulate the novel task of literary evidence retrieval , in which models are given an excerpt of literary analysis surrounding a masked quotation and asked to retrieve the quoted passage from the set of all passages in the work. Solving this retrieval task requires a deep understanding of complex literary and linguistic phenomena, which proves challenging to methods that overwhelmingly rely on lexical and semantic similarity matching. We implement a RoBERTa-based dense passage retriever for this task that outperforms existing pretrained information retrieval baselines; however, experiments and analysis by human domain experts indicate that there is substantial room for improvement over our dense retriever. When analyzing a literary work (e.g., a novel or short story), scholars make claims about the text and provide supporting evidence in the form of quotations from the work (Thompson, 2002; Finnegan, 2011; Graff et al., 2014). For example, Monaghan (1980) claims that Elizabeth, the main character in Jane Austen's Pride and Prejudice , doesn't just refuse an offer to join the standoffish bachelor Darcy and the wealthy Bingleys on their morning walk, but does so in such a way as to group Darcy with the snobbish Bingley sisters, and then directly quotes Elizabeth's tongue-in-cheek rejection: No, no; stay where you are. You are charmingly grouped, and appear to uncommon advantage. The picturesque would be spoilt by admitting a fourth. quotations (e.g., recognizing that Elizabeth says charm-ingly grouped and picturesque ironically in order to group Darcy with the snobbish Bingley sis-ters). This process requires a deep understanding of both literary phenomena, such as irony and metaphor, and linguistic phenomena (coreference, paraphrasing, and stylistics). In this paper, we computationally study the relationship between literary claims and quotations by collecting a large-scale dataset for R etrieving E vidence for Li terary C laims (RELiC), which contains 78K scholarly excerpts of literary analysis that each directly quote a passage from one of 79 widely-read English texts. The complexity of the claims and quotations in RELiC makes it a challenging testbed for modern neural retrievers: given just the text of the claim and analysis that surrounds a masked quotation, can a model retrieve the quoted passage from the set of all possible passages in the literary work? This literary evidence retrieval task (see Figure 1) differs considerably from retrieval problems commonly studied in NLP, such as those used for fact checking (Thorne et al., 2018), open-domain QA (Chen et al., 2017; Chen and Yih, 2020), and text generation (Krishna et al., 2021), in the relative lack of lexical or even semantic similarity between claims and queries. Instead of latching onto surface-level cues, our task requires models to understand complex devices in literary writing and apply general theories of interpretation. RELiC is also challenging because of the large number of retrieval candidates: for War and Peace , the longest literary work in the dataset, models must choose from one of  32K candidate passages. How well do state-of-the-art retrievers perform on RELiC? Inspired by recent research on dense passage retrieval (Guu et al., 2020; Karpukhin et al., 2020), we build a neural model (dense-RELiC) by embedding both scholarly claims and candidate literary quotations with pretrained RoBERTa networks (Liu et al., 2019), which are then fine-tuned 7500 Elizabeth comes to Pemberley full of fear of being treated as an interloper, a trespasser; even before any plans of visiting the ancient house are made, the mention of visiting Derbyshire makes Elizabeth feel like a thief: [masked quote] She seems to be afraid of encountering, if not the horrors of a Gothic castle, at least the resentment of a stern aristocrat It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. using a contrastive objective that encourages the representation for the ground-truth quotation to lie nearby to that of the claim. Both sparse retrieval methods such as BM25 and pretrained dense retrievers such as DPR and REALM perform poorly on RELiC, which underscores the difference between our dataset and existing information retrieval benchmarks (Thakur et al., 2021) on which these baselines are much more competitive. Our dense-RELiC model fares better than these baselines but still lags far behind human performance, and an analysis of its errors suggests that it struggles to understand complex literary phenomena. Finally, we qualitatively explore whether our dense-RELiC model can be used to support evidence-gathering efforts by researchers in the humanities. Inspired by prompt-based querying (Jiang et al., 2020), we issue our own out-of-distribution queries to the model by formulating simple descriptions of events or devices of interest (e.g., symbols of Gatsby's lavish lifestyle ) and discover that it often returns relevant quotations. To facilitate future research in this direction, we publicly release our dataset and models. 1 2 Collecting a Dataset for Literary Evidence Retrieval We collect a dataset for the task of R etrieving E vidence for Li terary C laims, or RELiC, the first large-scale retrieval dataset that focuses on the challenging literary domain. Each example in RELiC consists of two parts: (1) the context surround-1 https://relic.cs.umass.edu ing the quoted material, which consists of literary claims and analysis, and (2) a quotation from a widely-read English work of literature. This section describes our data collection and preprocessing, as well as a fine-grained analysis of 200 examples from RELiC to shed light on the types of quotations it contains. See Table 1 for corpus statistics. Selecting works of literature: We collect 79 primary source works written or translated into English 2 from Project Gutenberg and Project Gutenberg Australia. 3 These public domain sources were selected because of their popularity and status as members of the Western literary canon, which also yield more scholarship (Porter, 2018). All primary sources were published in America or Europe between 1811 and 1949. 77 of the 79 are fictional novels or novellas, one is a collection of short stories ( The Garden Party and Other Stories by Katherine Mansfield), and one is a collection of essays ( The Souls of Black Folk by W. E. B. Du Bois). Collecting quotations from literary analysis: We queried all documents in the HathiTrust Digital Library, 4 a collaborative repository of volumes from academic and research libraries, for exact matches of all sentences of ten or more tokens from each of the 79 works. The overwhelming majority 2 Of the 79 primary sources in RELiC, 72 were originally written in English, 3 were written in French, and 4 were written Russian. RELiC contains the corresponding English translations of these 7 primary source works. The complete list of primary source works is available in Appendix Tables A7, A8. 3 https://www.gutenberg.org/ 4 https://www.hathitrust.org/ 7501 # training examples 62,956 # validation examples 7,833 # test examples 7,785 # total examples 78,574 average context length (words) 157.7 average quotation length (words) 40.5 # primary sources 79 # unique sec. sources 8,836 Table 1: RELiC statistics. of HathiTrust documents are scholarly in nature, so most of these matches yielded critical analysis of the 79 primary source works. We received permission from the HathiTrust to publicly release short windows of text surrounding each matching quotation. Filtering and preprocessing: The scholarly articles we collected from our HathiTrust queries were filtered to exclude duplicates and non-English sources. We then preprocessed the resulting text to remove pervasive artifacts such as in-line citations, headers, footers, page numbers, and word breaks using a pattern-matching approach (details in Appendix A). Finally, we applied sentence tokenization using spaCy's dependency parser-based sentence segmenter 5 to standardize the size of the windows in our dataset. Each window in RELiC contains the identified quotation and four sentences of claims and analysis 6 on each side of the quotation (see Table 2 for examples). To avoid asking models to retrieve a quote they have already seen during training, we create training, validation, and test splits such that primary sources in each fold are mutually exclusive. Statistics of our dataset sources are provided in Appendix A.3. Table 1 contains detailed statistics of RELiC. To the best of our knowledge, RELiC is the first retrieval dataset in the literary domain, and the only 5 https://spacy.io/ , the default segmenter in spaCy is modified to use ellipses, colons, and semicolons as custom sentence boundaries, based on the observation that literary scholars often only quote part of what would typically be defined as a sentence. 6 The HathiTrust permitted us to release windows consisting of up to eight sentences of scholarly analysis. While more context is of course desirable, we note that (1) conventional model sizes are limited in input sequence length, and (2) context further away from the quoted material has diminishing value, as it is likely to be less relevant to the quoted span. one that requires understanding complex phenomena like irony and metaphor. We provide a detailed comparison of RELiC to other retrieval datasets in the recently-proposed BEIR retrieval benchmark (Thakur et al., 2021) in Appendix Table A6. RELiC has a much longer query length (157.7 tokens on average) than all BEIR datasets except Ar-guAna (Wachsmuth et al., 2018). Furthermore, our results in Section 3.3 show that while these longer queries confuse pretrained retriever models (which heavily rely on token overlap), a model trained on RELiC is able to leverage the longer queries for better retrieval. What are the different ways in which literary scholars use direct quotation in RELiC? We perform a manual analysis of 200 held-out examples to gain a better understanding of quotation usage, categorizing each quotation into the following three types: Claim-supporting evidence: In 151 of the 200 annotated examples, literary scholars used direct quotation to provide evidence for a more general claim about the primary source work. In the first row of Table 2, Hartstein (1985) claims that this whale... brings into focus such fundamental questions as the knowability of space: and then quotes the following metaphorical description from Moby Dick as evidence: And as for this whale spout, you might almost stand in it, and yet be undecided as to what it is precisely. When quoted material is used as claim-supporting evidence , the context before and after usually refers directly to the quoted material; 7 for example, the paradoxes of reality and uncertainties of this world are exemplified by the vague nature of the whale spout. Paraphrase-supporting evidence: In 31 of the examples, we observe that scholars used the primary source work to support their own paraphrasing of the plot in order to contextualize later analysis. In the second row of Table 2, Blackstone (1972) uses the quoted material to enhance a summary of a specific scene in which Jacob's mind is wandering during a chapel service. Jacob's daydreaming is later used in an analysis of Cambridge as a location in Virginia Woolf's works, but no literary argument is made in the immediate context. When quoted material is being employed as 7 In 19 of the 151 claim-supporting evidence examples, scholars introduce quoted material by explicitly referring to a specific sentence, passage, scene, or similar delineation. Miscellaneous: 18 of the 200 samples were not literary analysis, though some were still related to literature (for example, analysis of the the film adaptation of The Age of Innocence ). Others were excerpts from the primary sources that suffered from severe OCR artifacts and were not detected or extracted by the methods in Appendix A.2. Having established that the examples in RELiC contain complex interplay between literary quotation and scholarly analysis, we now shift to measuring how well neural models can understand these interactions. In this section, we first formalize our evidence retrieval task, which provides the scholarly context without the quotation as input to a model, along with a set of candidate passages that come from the same book, and asks the model to retrieve the ground-truth missing quotation from the candidates. Then, we describe standard information retrieval baselines as well as a RoBERTa-based ranking model that we implement to solve our task. Formally, we represent a single window in RELiC from book b as ( ..., l  2 , l  1 , q n , r 1 , r 2 , ... ) where q n is the quoted n -sentence long passage, and l i and r j correspond to individual sentences before and after the quotation in the scholarly article, respectively. The window size on each side is bounded by hyperparameters l max and r max , each of which can be up to 4 sentences. Given the l  l max :  1 and r 1: r max sentences surrounding the missing quotation, we ask models to identify the quoted passage q n from the candidate set C b,n , which consists of all n -sentence long passages in book b (see Figure 1). This is a particularly challenging retrieval task because the candidates are part of the same overall narrative and thus mention the same overall set of entities (e.g., characters, locations) and other plot elements, which is a disadvantage for methods based on string overlap. Evaluation: Models built for our task must produce a ranked list of candidates C b,n for each example. We evaluate these rankings using both recall@ k for k = 1 , 3 , 5 , 10 , 50 , 100 and mean rank of q in the ranked list. Both types of metrics focus on the position of the ground-truth quotation 7503 Model L/R Recall@ k (  ) Avg rank (  ) Proxy task acc (  ) 1 3 5 10 50 100 (non-parametric / pretrained zero-shot) random 0.0 0.1 0.1 0.2 1.2 2.5 2445.1 33.3 BM25 1/1 1.2 3.2 4.2 5.9 12.5 17.0 1561.2  9 BM25 4/4 1.3 2.9 4.1 6.7 14.5 19.7 1386.8  SIM (Wieting et al., 2019) 1/1 1.3 2.8 3.8 5.6 13.4 18.8 1350.0 23.0 SIM (Wieting et al., 2019) 4/4 0.9 2.1 3.0 4.7 12.2 17.3 1358.2 11.0 DPR (Karpukhin et al., 2020) 1/1 1.3 3.0 4.3 6.6 15.4 22.2 1205.3 25.5 DPR (Karpukhin et al., 2020) 4/4 1.0 2.2 3.2 5.2 13.9 20.7 1208.1 22.5 c-REALM (Krishna et al., 2021) 1/1 1.6 3.5 4.8 7.1 15.9 21.7 1332.0 23.0 c-REALM (Krishna et al., 2021) 4/4 0.9 2.1 3.3 5.0 12.9 18.8 1333.9 17.5 ColBERT (Khattab and Zaharia, 2020) 1/1 2.9 6.0 7.8 11.0 21.4 27.9 N/A 8 38.8 ColBERT (Khattab and Zaharia, 2020) 4/4 1.9 3.9 5.3 8.0 18.2 25.2 N/A 18.9 (trained on RELiC training set) dense-RELiC 0/1 3.4 7.1 9.3 12.6 24.1 31.3 1094.4 42.5 0/4 5.2 10.7 13.6 18.5 32.4 40.2 887.8 46.5 1/0 5.2 10.5 13.6 18.7 34.7 43.2 788.5 67.5 4/0 6.8 14.4 19.3 25.7 43.9 52.8 538.3 65.5 1/1 7.8 15.1 19.3 25.7 43.3 52.0 558.0 67.0 4/4 9.4 18.3 24.0 32.4 51.3 60.8 377.3 65.0 Human domain experts 4/4 93.5 Table 3: Overall comparison of different systems and context sizes (L/R indicates the number of sentences on the left and right side of the missing quote) on the test set of RELiC using recall@ k metrics, normalized to a maximum score of 100. q in the ranked list, and neither gives special treatment to candidates that overlap with q . As such, recall@1 alone is overly strict when the quotation length l > 1 , which is why we show recall at multiple values of k . An additional motivation is that there may be multiple different candidates that fit a single context equally well. We also report accuracy on a proxy task with only three candidates, which allows us to compare with human performance as described in Section 4. 3.2 Models Baselines: Our baselines include both standard term matching methods as well as pretrained dense retrievers. BM25 (Robertson et al., 1995) is a bag-of-words method that is very effective for information retrieval. We form queries by concatenating the left and right context and use the implementation from the rank_bm25 library 10 to build a BM25 model for each unique candidate set C b,n , tuning 8 ColBERT does not provide a ranking for candidates outside the top 1000, so we cannot report mean rank. 9 We do not report BM25's accuracy on the proxy task because its top-ranked quotes were used as candidates in the proxy task in addition to the ground-truth quotation. the free parameters as per Kamphuis et al. (2020). 11 Meanwhile, our dense retrieval baselines are pretrained neural encoders that map queries and candidates to vectors. We compute vector similarity scores (e.g., cosine similarity) between every query/candidate pair, which are used to rank candidates for every query and perform retrieval. We consider the following four pretrained dense retriever baselines in our work, which we deploy in a zero-shot manner (i.e., not fine-tuned on RELiC):  DPR (Dense Passage Retrieval) is a dense retrieval model from Karpukhin et al. (2020) trained to retrieve relevant context paragraphs in open-domain question answering. We use the DPR context encoder 12 pretrained on Natural Questions (Kwiatkowski et al., 2019) with dot product as a similarity function. SIM is a semantic similarity model from Wieting et al. (2019) that is effective on semantic textual similarity benchmarks (Agirre et al., 2016). SIM is trained on ParaNMT (Wiet-ing and Gimpel, 2018), a dataset containing 11 We set k 1 = 0.5, b = 0.9 after tuning on validation data. 12 https://huggingface.co/facebook/ dpr-ctx_encoder-single-nq-base 7504 16.8M paraphrases; we follow the original implementation, 13 and use cosine similarity as the similarity function. c-REALM (contrastive Retrieval Augmented Language Model) is a dense retrieval model from Krishna et al. (2021) trained to retrieve relevant contexts in open-domain long-form question answering, and shown to be a better retriever than REALM (Guu et al., 2020) on the ELI5 KILT benchmark (Fan et al., 2019; Petroni et al., 2021). ColBERT is a ranking model from Khattab and Zaharia (2020) that estimates the relevance between a query and a document using contextualized late interaction. It is trained on MS MARCO ranking data (Nguyen et al., 2016). Training retrievers on RELiC (dense-RELiC): Both BM25 and the pretrained dense retriever baselines perform similarly poorly on RELiC (Table 3). These methods are unable to capture more complex interactions within RELiC that do not exhibit extensive string overlap between quotation and context. As such, we also implement a strong neural retrieval model that is actually trained on RELiC, using a similar setup to DPR and REALM. We first form a context string c by concatenating a window of sentences on either side of the quotation q (replaced by a MASK token), c = ( l  l max , ..., l  1 , [MASK] , r 1 , ..., r r max ) We train two encoder neural networks to project the literary context and quote to fixed 768d vectors. Specifically, we project c and q using separate encoder networks initialized with a pretrained RoBERTa-base model (Liu et al., 2019). We use the <s> token of RoBERTa to obtain 768d vectors for the context and quotation, which we denote as c i and q i . To train this model, we use a contrastive objective (Chen et al., 2020) that pushes the context vector c i close to its quotation vector q i , but away from all other quotation vectors q j in the same minibatch (in-batch negative sampling): loss =  (cid:88) ( c i ,q i )  B log exp c i  q i (cid:80) q j  B exp c i  q j 13 https://github.com/jwieting/ beyond-bleu where B is a minibatch. Note that the size of the minibatch | B | is an important hyperparameter since it determines the number of negative samples. 14 All elements of the minibatch are context/quotation pairs sampled from the same book. During inference, we rank all quotation candidate vectors by their dot product with the context vector. We report results from the baselines and our dense-RELiC model in Table 3 with varying context sizes where L/R refers to L preceding context sentences and R subsequent context sentences. While all models substantially outperform random candidate selection, all pretrained neural dense retrievers perform similarly to BM25, with ColBERT being the best pretrained neural retriever (2.9 recall@1). This result indicates that matching based on string overlap or semantic similarity is not enough to solve RELiC, and even powerful neural retrievers struggle on this benchmark. Training on RELiC is crucial: our best-performing dense-RELiC model performs 7x better than BM25 (9.4 vs 1.3 recall@1). performance: Table 3 shows that dense-RELiC effectively utilizes longer context  feeding only one sentence on each side of the quotation (1/1) is not as effective as a longer context (4/4) of four sentences on each side (7.8 vs 9.4 recall@1). However, the longer contexts hurt performance for pretrained dense retrievers in the zero-shot setting (1.6 vs 0.9 recall@1 for c-REALM), perhaps because context further away from the quotation is less likely to be helpful. Finally, we observe that dense-RELiC performance is strictly better (5.2 vs 6.8 recall@1) when the model is given only preceding context (4/0 or 1/0) compared to when the model is given only subsequent context (0/4 or 0/1). Dense vs. sparse retrievers: As expected, BM25 retrieves the correct quotation when there is significant string overlap between the quotation and context, as in the following example from The Great Gatsby , in which the terms sky , bloom , Mrs. McKee , voice , call , and back appear in both places: 14 We set | B | = 100 , and train all models for 10 epochs on a single RTX8000 GPU with an initial learning rate of 1e-5 using the Adam optimizer (Kingma and Ba, 2015), early stopping on validation loss. Models typically took 4 hours to complete 10 epochs. Our implementation uses the Hugging-Face transformers library (Wolf et al., 2020). The total number of model parameters is 249M. Yet his analogy also implicitly unites the two women. Myrtle's expansion and revolution in the smoky air are also outgrowths of her surreal attributes, stemming from her residency in the Valley of Ashes. The late afternoon sky bloomed in the window for a moment like the blue honey of the Mediterranean-then the shrill voice of Mrs. McKee called me back into the room. The objective talk of Monte Carlo and Marseille has made Nick daydream. In Chapter I Daisy and the rooms had bloomed for him, with him, and now the sky blooms. The fact that Mrs. McKee's voice calls him back clearly reveals the subjective daydreamy nature of this statement. However, this behavior is undesirable for most examples in RELiC, since string overlap is generally not predictive of the relationship between quotations and claims. The top row of Table 5 contains one such example, where dense-RELiC correctly chooses the missing quotation while BM25 is misled by string overlap. How well do humans actually perform on RELiC? To compare the performance of our dense retriever to that of humans, we hired six domain experts with at least undergraduate-level degrees in English literature from the Upwork 15 freelancing platform. Because providing thousands of candidates to a human evaluator is infeasible, we instead measure human performance on a simplified proxy task: we provide our evaluators with four sentences on either side of a missing quotation from Pride and Prejudice 16 and ask them to select one of only three candidates to fill in the blank. We obtain human judgments both to measure a human upper bound on this proxy task as well as to evaluate whether humans struggle with examples that fool our model. Human upper bound: First, to measure a human upper bound on this proxy task, we chose 200 test set examples from Pride and Prejudice and formed a candidate pool for each by including BM25's top two ranked answers along with the ground-truth quotation for the single sentence case. As the task is trivial to solve with random candidates, we decided to use a model to select harder negatives, and we chose BM25 to see if humans would be distracted by high string overlap in the negatives. Each of the 200 examples was separately annotated by three experts, and they were 15 https://upwork.com 16 We decided to keep our proxy task restricted to the most well-known book in our test set because of the ease with which we could find highly-qualified workers who self-reported that they had read (and often even re-read) Pride and Prejudice . paid $100 for annotating 100 examples. The last column of Table 3 compares all of our baselines along with dense-RELiC against human domain experts on this proxy task. Humans substantially outperform all models on the task, with at least two of the three domain experts selecting the correct quote 93.5% of the time; meanwhile, the highest score for dense-RELiC is 67.5%, which indicates huge room for improvement. Interestingly, all of the zero-shot dense retrievers except ColBERT 1/1 underperform random selection on this task; we theorize that this is because all of these retrievers are misled by the high string overlap of the negative BM25-selected examples. Table 4 confirms substantial agreement among our annotators. Three domain experts attempted 100 of these examples and achieved an accuracy of 94%, demonstrating that humans can easily disambiguate cases on which our model fails, though we note our model's poorer performance when retrieving a single sentence (as in the proxy task) versus multiple sentences (A5). The bottom two rows of Table 5 contain instances in which all human annotators agreed on the correct candidate but dense-RELiC failed to rank it in the top 1000. In one, all human annotators immediately recognized the opening line of Pride and Prejudice , one 17 In our proxy task each instance has a different set of candidate quotations, which we randomly shuffle before showing annotators. Since our task is not strictly categorical, while computing Fleiss Kappa we define category as the option number shown to annotators. We believe this definition is closest to the free-marginal nature of our task (Randolph, 2010). of the most famous in English literature. In the other, the claim mentions that the interpretation hinges on a single word's (got) connotation of a market, which humans understood. Issuing out-of-distribution queries to the retriever: Does our dense-RELiC model have potential to support humanities scholars in their evidence-gathering process? Inspired by prompt-based learning, we manually craft simple yet out-of-distribution prompts and queried our dense-RELiC retriever trained with 1 sentence of left context and no right context. A qualitative inspection of the top-ranked quotations in response to these prompts (Table 6) reveals that the retriever is able to obtain evidence for distinct character traits, such as the ignorance of the titular character in Frankenstein or Gatsby's wealthy lifestyle in The Great Gatsby . More impressively, when queried for an example from Pride and Prejudice of the main character, Elizabeth, demonstrating frustration towards her mother, the retriever returns relevant excerpts in the first-person that do not mention Elizabeth, and the top-ranked quotations have little to no string overlap with the prompts. Limitations: While these results show dense-RELiC's potential to assist research in the humanities, the model suffers from the limited expressivity of its candidate quotation embeddings q i , and addressing this problem is an important direction for future work. The quotation embeddings do not incorporate any broader context from the narrative, which prevents resolving coreferences to pronominal character mentions and understanding other important discourse phenomena. For example, Table A5 shows that dense-RELiC 's top two 1-sentence candidates for the above Pride and Prejudice example are not appropriate evidence for the literary claim; the increased relevancy of the 2-sentence candidates (Table 6, third row) over the 1-sentence candidates suggests that dense-RELiC may benefit from more contextualized quotation embeddings. Furthermore, dense-RELiC struggles with retrieving concepts unique to a text, such as the hypnopaedic phrases strewn throughout Brave New World (Table 6, bottom). such as LitBank (Bamman et al., 2019; Sims et al., 2019), an annotated dataset of 100 works of fic-tion with annotations of entities, events, coreferences, and quotations. Papay and Pad (2020) introduced RiQuA, an annotated dataset of quotations in English literary text for studying dialogue structure, while Chaturvedi et al. (2016) and Iyyer et al. (2016) characterize character relationships in novels. Our work also relates to quotability identification (MacLaughlin and Smith, 2021), which focuses on ranking passages in a literary work by how often they are quoted in a larger collection. Unlike RELiC, however, these datasets do not contain literary analysis about the works. Retrieving cited material: Citation retrieval closely relates to RELiC and has a long history of research, mostly on scientific papers: O'Connor (1982) formulated the task of document retrieval using citing statements, which Liu et al. (2014) revisit to create a reference retrieval tool that recommends references given context. Bertin et al. (2016) examine the rhetorical structure of citation contexts. Perhaps closest to RELiC is the work of Grav (2019), which concentrates on the quotation of secondary sources in other secondary sources, unlike our focus on quotation from primary sources. Finally, as described in more detail in Section 2.2 and Appendix A6, RELiC differs significantly from existing NLP and IR retrieval datasets in domain, linguistic complexity, and query length. In this work, we introduce the task of literary evidence retrieval and an accompanying dataset, RELiC. We find that direct quotation of primary sources in literary analysis is most commonly used as evidence for literary claims or arguments. We train a dense retriever model for our task; while it significantly outperforms baselines, human performance indicates a large room for improvement. Important future directions include (1) building better models of primary sources that integrate narrative and discourse structure into the candidate representations instead of computing them out-of-context, and (2) integrating RELiC models into real tools that can benefit humanities researchers. First and foremost, we would like to thank the HathiTrust Research Center staff (especially Ryan Dubnicek) for their extensive feedback throughout our project. We are also grateful to Naveen Jafer Nizar for his help in cleaning the dataset, Vishal Kalakonnavar for his help with the project web-page, Marzena Karpinska for her guidance on computing inter-annotator agreement, and the UMass NLP community for their insights and discussions during this project. KT and MI are supported by awards IIS-1955567 and IIS-2046248 from the National Science Foundation (NSF). KK is supported by the Google PhD Fellowship awarded in 2021. We acknowledge that the group of authors from whom we selected primary sources lacks diversity because we selected from among digitized, public domain sources in the Western literary canon, which is heavily biased towards white, male writers. We made this choice because there are relatively few primary sources in the public domain that are written by minority authors and also have substantial amounts of literary analysis written about them. We hope that our data collection approach will be followed by those with access to copyrighted texts in an effort to collect a more diverse dataset. The experiments involving humans were reviewed by the UMass Amherst IRB with a status of Exempt.
abstain objective abstain result abstain other abstain abstain method abstain other method abstain abstain method abstain method abstain objective method abstain objective other abstain abstain method abstain abstain abstain method abstain abstain abstain other abstain abstain method abstain method abstain abstain method abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method other abstain abstain abstain method abstain abstain method method abstain abstain method method method abstain method other abstain abstain abstain abstain abstain method abstain method abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain other abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain result result abstain other other other other abstain abstain method abstain