It is very common to use quotations (quotes) to make our writings more elegant or convincing. To help people find appropriate quotes efficiently, the task of quote recommendation is presented, aiming to recommend quotes that fit the current context of writing. There have been various quote recommendation approaches, but they are evaluated on different unpublished datasets. To facilitate the research on this task, we build a large and fully open quote recommendation dataset called QuoteR, which comprises three parts including English, standard Chinese and classical Chinese. Any part of it is larger than previous unpublished counterparts. We conduct an extensive evaluation of existing quote recommendation methods on QuoteR. Furthermore, we propose a new quote recommendation model that significantly outperforms previous methods on all three parts of QuoteR. All the code and data of this paper can be obtained at https://github.com/ thunlp/QuoteR . A quotation, or quote for short, is a sequence of words that someone else has said or written. Quotes, especially the famous quotes including proverbs, maxims and other famous sayings, are quite useful in writing  they can not only help illuminate and emphasize the meaning we want to convey, but also endow our writing with elegance and credibility (Cole, 2008). As a result, the use of quotes is very common and, moreover, universal among all languages. However, it is not an easy job for ordinary people to promptly come up with appropriate quotes that fit the current context of writing, due to the huge number of quotes. Search engines can provide some help in finding quotes by keyword matching, but it is often not enough. Quotes generally  Equal contribution  Corresponding author. There's an old Bible verse my dad used to say all the time that says sufficient unto the day is the evil thereof , Pyron said. In other words  today has its own set of problems, we can't do anything about yesterday, and I don't want to jump too far into tomorrow. express their meanings implicitly by rhetorical devices like metaphor and have different word usages from modern and everyday writing, as illustrated in Figure 1, for which quote search based on keyword matching is ineffective. In addition, some quote repository websites organize quotes by topic. However, even after filtering by topic, there are still too many candidate quotes, and selecting a suitable one remains time-consuming. To tackle these challenges, Tan et al. (2015) introduce the task of quote recommendation, aiming to automatically recommend suitable quotes given the context of writing. 1 Afterward, a series of studies propose various approaches to this task (Ahn et al., 2016; Tan et al., 2016, 2018). However, these studies use different evaluation datasets, and none of them are publicly available. The lack of a standard and open dataset is undoubtedly a serious obstacle to the quote recommendation research. In this paper, to solve this problem, we build a large quote recommendation dataset that is publicly available. This dataset is named QuoteR (abbreviated from Quote R ecommendataion) and composed of three parts: (1) the English part that comprises 6,108 English quotes with 126,713 contexts; (2) the standard Chinese (Mandarin) part, which contains 3,004 standard Chinese quotes with 40,842 contexts; and (3) the classical Chinese (Wenyan) part, which comprises 4,438 classical Chinese quotes (including classical poems) and 116,537 contexts. Any part of this dataset is abso-1 This task also has great value to research, as a touchstone for NLP models' abilities in language understanding, semantic matching and linguistic coherence estimation. We conduct a fair and extensive evaluation of existing quote recommendation methods on QuoteR with a thorough set of metrics. By analyzing these methods and their evaluation results, we find two weaknesses of these methods and propose a new method by making corresponding improvements, which we believe would serve as a strong baseline for quote recommendation. First, most existing methods encode contexts and quotes into vectors for quote-context matching, using LSTM (Hochreiter and Schmidhuber, 1997) or CNN (Kim, 2014) as the encoders. These encoders have proven inferior to the pre-trained language models like BERT (Devlin et al., 2019), which limits the final quote recommendation performance. Therefore, we try to utilize a pre-trained language model, specifically BERT, as the sentence encoders to learn representations of quotes and contexts. Considering the huge compute resulting from the large scale of the dataset and the BERT model, it is nontrivial to train the context and quote encoders simultaneously. We design an ingenious training strategy to address this issue. Second, it is harder to learn good representations for quotes compared with contexts, because most quotes are quite pithy, and their words usually carry rich semantics, as shown in Figure 1. Existing methods, however, do not address this challenge well. To handle this challenge, we incorporate a kind of general lexical knowledge, namely sememes , into the quote encoder, aiming to improve the representations of quotes. A sememe is defined as the minimum semantic unit in linguistics (Bloomfield, 1926), and the sememes of a word atomically interpret the meaning of the word. Incorporating sememes can bring more semantic information for quote representation learning and conduce to a better quote vector. In experiments, we demonstrate that both the utilization of BERT and the incorporation of sememes substantially improve quote recommendation performance. And the sememe-incorporated BERT-based model significantly outperforms all previous methods on QuoteR. Moroever, ablation and case studies as well as human evaluation further prove its effectiveness. To conclude, our contributions are threefold: (1) building a large and the first open quote recommendation dataset; (2) conducting an extensive and fair evaluation of existing quote recommendation methods; (3) proposing a quote recommendation model that outperforms all previous methods and can serve as a strong baseline for future research. The task of quote recommendation is originally presented in Tan et al. (2015). They propose a learning-to-rank framework for this task, which integrates 16 hand-crafted features. Tan et al. (2016) and Tan et al. (2018) introduce neural networks to the quote recommendation task. They use LSTMs to learn distributed vector representations of contexts and quotes and conduct sentence matching with these vectors. Ahn et al. (2016) combine four different quote recommendation approaches including matching granularity adjustment (a statistical context-quote relevance prediction method), random forest, CNN and LSTM. In addition quote recommendation for writing, some studies focus on recommending quotes in dialog. Lee et al. (2016) propose an LSTM-CNN combination model to recommend quotes according to Twitter dialog threads, i.e., sequences of linked tweets. Wang et al. (2020) utilize an encoder-decoder framework to generate quotes as response, based on the separate modeling of the dialog history and current query. Wang et al. (2021) adopt a semantic matching fashion, which encodes the multiturn dialog history with Transformer (Vaswani et al., 2017) and GRU (Cho et al., 2014) and encodes the quote with Transformer. In terms of the datasets of quote recommendation for writing, Tan et al. (2015) construct an English dataset comprising 3,158 quotes and 64,323 contexts extracted from e-books in Project Gutenberg. 2 Ahn et al. (2016) build a similar English dataset that contains 400 most frequent quotes with contexts from e-books in Project Gutenberg and blogs. Tan et al. (2018) build a classical Chinese poetry quotation dataset that comprises over 9,000 poem sentences with 56,949 contexts extracted from Chinese e-books on the Internet. Unfortunately, all these datasets are not publicly available. Quote recommendation is essentially a kind of content-based recommendation task (Pazzani and Billsus, 2007), which is aimed at recommending products to users according to product descriptions and users' profiles. A closely related and widely studied task is content-based citation recommendation (Strohman et al., 2007), especially local citation recommendation that recommends related papers given a particular context of academic writing (He et al., 2010; Huang et al., 2012, 2015). Compared with quote recommendation, this task is targeted at structured documents (papers), which are much longer and possess abundant information such as title, abstract and citation relations that are useful for recommendation. Quotes are shorter and usually have no available information except the text, which renders quote recommendation more challenging. Another highly related but niche task is idiom recommendation (Liu et al., 2018, 2019), which aims to recommend appropriate idioms for a given context. Existing idiom recommendation methods are essentially covered by the quote recommendation methods described in 2.1. Liu et al. (2018) recommend idioms by learning representations of the contexts and idioms, similar to the context-quote relevance-based quote recommendation methods (Ahn et al., 2016; Tan et al., 2018). The difference lies in the use of word embeddings of idioms rather than a sentence encoder. Liu et al. (2019) regard idiom recommendation as a context-to-idiom machine translation problem and use an LSTM-based encoder-decoder framework, which is similar to Wang et al. (2020). In addition to quote recommendation, there are some other quote-related tasks. For example, quote detection (or recognition) that is aimed at locating spans of quotes in text (Pouliquen et al., 2007; Scheible et al., 2016; Pareti et al., 2013; Papay and Pad, 2019), and quote attribution that intends to automatically attribute quotes to speakers in the text (Elson and McKeown, 2010; O'Keefe et al., 2012; Almeida et al., 2014; Muzny et al., 2017). Different from quote recommendation that focuses on famous quotes, these tasks mainly deal with the general quotes of utterance. Before describing our dataset and model, we first formulate the task of quote recommendation for writing and introduce several basic concepts, most of which follow previous work (Tan et al., 2015). For a piece of text containing a quote q , the text segment occurring before the quote is named left context c l while the text segment occurring after the quote is named right context c r . The concatenation of left and right contexts form the quote context c = [ c l ; c r ] . Suppose there is a quote set that comprises all the known candidate quotes Q = { q 1 ,    , q | Q | } , where |  | denotes the cardinality of a set. In the task of quote recommendation for writing, a query context c = [ c l ; c r ] is given, and the gold quote q c is wanted, where the query context is the context provided by the user and the gold quote is the quote in the quote set that fits the query context best. Theoretically, a query context may have more than one gold quote because there are some quotes that convey almost the same meaning. Following previous work (Tan et al., 2015; Ahn et al., 2016), for simplicity, we only regard the quote that actually appears together with the query context in corpora as the gold quote. For a quote recommendation model, given the quote set Q , its input is a query context c = [ c l ; c r ] , and it is supposed to calculate a rank score for each candidate quote in Q and output a quote list according to the descending rank scores. In this section, we present the building process and details of the QuoteR dataset. We begin with the English part. We choose the popular and free quote repository website Wikiquote 3 as the source of English quotes. We download its official dump and extract over 60,000 English quotes in total to form the quote set. We notice that previous work (Tan et al., 2015; Ahn et al., 2016) collects quotes from another website named Library of Quotes, but this website has closed down. To obtain real contexts of quotes, we use three corpora. The first is the Project Gutenberg corpus that previous studies use, which comprises over 50,000 e-books. The second corpus is BookCorpus containing about 11,000 e-books (Zhu et al., 2015). In addition to the two book corpora, we use the OpenWebText corpus (Gokaslan and Cohen, 2019) which is composed of text from web pages and has different text styles from books. The total size of the raw text of the three corpora reaches 48.8 GB. We search all the corpora for the occurrences of quotes in the quote set. Some quotes are composed of multiple sentences, and only part of them are cited in some cases. To cope with this situation, we split each quote into sentences using Stanza (Qi et al., 2020) and then search for each constituent sentence in the corpora. If multiple constituent sentences of a quote appear sequentially, we combine them into an occurrence of the quote. Compared with previous work that searches for quotes as a whole (Tan et al., 2015; Ahn et al., 2016), we can find more quote occurrences. For each quote occurrence, we take the 40 words preceding and following it as its left and right contexts, respectively. The concatenation of the left and right contexts forms a context, and a context and the corresponding quote form a context-quote pair. We remove the repeated context-quote pairs and filter out the quotes appearing less than 5 times in the corpora. To avoid dataset imbalance, we randomly select 200 context-quote pairs for a quote appearing more than 200 times and discard its other context-quote pairs. Finally, we obtain 126,713 context-quote pairs involving 6,108 different quotes, which form the English part of QuoteR. We split all the context-quote pairs into training, validation and test sets roughly in the ratio 8:1:1, making sure that all the quotes appear in the validation and test sets while 100 quotes do not appear in the training set. We split the dataset in this way in order to observe how quote recommendation models perform in the zero-shot situation, where the model has never seen the gold quote of some vali-dation/test contexts during training. The statistics of the final split dataset are listed in Table 1. 4.2 The Standard Chinese Part We gather standard Chinese quotes from a large quote collection website named Juzimi 4 . More than 32,000 standard Chinese quotes are collected 4 https://www.juzimi.com/ altogether. To obtain quote contexts, we use two corpora including a corpus composed of answer text from a Chinese QA website 5 and a large-scale book corpus that we specifically build and comprises over 8,000 free Chinese e-books. The total size of the two corpora is about 32 GB. Then we use the same method in building the English part to extract quote occurrences from the corpora. Since Chinese is not naturally word-segmented, we take the 50 characters (rather than words) before and after a quote occurrence as the left and right contexts. In addition, since there are fewer quotes and contexts for the standard Chinese part, we reduce the minimum number of occurrences for a selected quote to 3, and the maximum number of retained contexts per quote to 150. After deduplication and filtering, we obtain the standard Chinese part of QuoteR, which has 40,842 context-quote pairs involving 3,004 quotes. We split the standard Chinese part in the same way as the English part, and the statistics are also shown in Table 1. 4.3 The Classical Chinese Part Classical Chinese quotes, including classical poems and proverbs, are often cited in standard Chinese writing. Considering that classical Chinese is very different from standard Chinese, we separate classical Chinese quotes from standard Chinese ones. We collect over 17,000 classical Chinese quotes from Gushiwenwang, 6 a classical Chinese poetry and literature repository website, and aforementioned Juzimi. 7 Then we adopt the same way as standard Chinese to extract context-quote pairs from the two Chinese corpora and conduct deduplication and filtering. Finally, we obtain the classical Chinese part of QuoteR that comprises 116,537 context-quote pairs of 4,438 quotes. The statistics of this part after splitting are also in Table 1. 4.4 Quality Assessment by Human After the construction of QuoteR, we assess its quality by human. For each part, we randomly sample 100 context-quote pairs, and ask three annotators to independently determine whether each quote fits the corresponding context. The final results are 5 https://github.com/brightmart/nlp_ chinese_corpus 6 https://www.gushiwen.org/ 7 Juzimi provides the dates when the quotes appear so that we can distinguish classical and standard Chinese quotes. obtained by voting. Finally, 99/98/94 context-quote pairs are regard as suitable for the three parts, respectively. The results verify the quality of QuoteR, which is expected because the data are extracted from high-quality corpora like books. In this section, we elaborate on our proposed quote recommendation model. This model is based on the representative pre-trained language model BERT (Devlin et al., 2019), but can be readily adapted to other pre-trained language models. Similar to most previous methods (Tan et al., 2016; Ahn et al., 2016), we use BERT as the text encoder to learn vector representations of contexts and quotes, and then calculate the similarity between the representations of the query context and a candidate quote as the rank score of the quote. We first obtain the representations of quotes. Formally, for a candidate quote comprising m tokens q = { x 1 ,    , x m }  Q , we feed it into BERT and obtain a series of hidden states: h q [C] , h q 1 ,    , h q m = BERT q ( [C] , x 1 ,    , x m ) , (1) where [C] denotes the special [CLS] token in BERT that is added to the front of a sequence. Following Devlin et al. (2019), we use the hidden state of [C] as the representation of the quote: q = h q [C] . The representations of all quotes form the quote representation matrix Q = [ q 1 ,    , q | Q | ]. We can use another BERT as the context encoder to obtain the representation of the query context c = [ c l ; c r ] . Considering the context is composed of left and right contexts that are not naturally joined, we can insert an additional separator token between them before feeding them into BERT: h c [C] ,    = BERT c ( [C] , c l , [S] , c r ) , (2) where [S] is the sentence separator token [SEP] in BERT. We can also use the hidden state of [C] as the representation of the context: c = h c [C] . However, it is actually inconsistent with the general use of BERT. Whether in pre-training or fine-tuning, when the input to BERT is two text segments connected by the separator token, the hidden state of [CLS] is only used to classify the relation between the two segments, e.g., to predict whether the second segment is the actual next sentence of the first segment in the next sentence prediction (NSP) pre-training task (Devlin et al., 2019). We turn to another pre-training task of BERT, masked language modeling (MLM), which is a cloze task (Taylor, 1953) aimed at predicting masked tokens. Specifically, some tokens in a text sequence are randomly substituted by the special [MASK] tokens and the hidden states of the [MASK] tokens are fed into a classifier to predict the original tokens. Quote recommendation given context can be regarded as a special cloze task whose object of prediction is quotes rather than tokens. Inspired by the MLM pre-training task, we propose another way to learn the context representation by inserting an additional [MASK] token: h c [C] ,    , h c [M] ,    = BERT c ( [C] , c l , [M] , c r ) , (3) where [M] is the [MASK] token. We use the hidden state of [M] as the representation of the query context: c = h c [M] . 8 Calculating Rank Scores of Candidate Quotes After obtaining the representations of all candidate quotes and the query context, the rank score of a candidate quote can be calculated by softmax: p = softmax( Q  c ) , (4) where p is a normalized probability vector whose i -th element is the rank score of the i -th quote. As in previous work (Tan et al., 2016), we can simply use the cross-entropy loss to train the quote and context encoders simultaneously. However, there are two problems. (1) For each context in the training set, the quote encoder needs to be updated for every quote in the quote set. In other words, the BERT-based quote encoder would be fine-tuned thousands of times per training instance, which requires formidably big GPU memory and long training time. 9 (2) The huge imbalance between positive and negative samples (one vs. several thousands) would weaken the capacity of the 8 The hidden state of [M] can also be regarded as the representation of the required quote for the query context. In this view, the rank score in Eq. (4) is actually calculated by the similarity between a candidate quote and the required quote. 9 We find that four 16-GB GPUs would be out of memory during training even though we set the batch size to 1. 340 quote encoder and, in turn, impair the final quote recommendation performance. A simple solution is to freeze the quote encoder during training, i.e., use the raw pre-trained BERT as the quote encoder, and train the context encoder only. But the untrained quote encoder would decrease final quote recommendation performance, as demonstrated in later experiments. To address these issues, inspired by the study on noise contrastive estimation (NCE) (Gutmann and Hyvrinen, 2012), we adopt the negative sampling strategy in training. For each context-quote pair, we select some non-gold quotes as negative samples, and calculate a pseudo-rank score of the gold quote among the selected quotes. Formally, for a context-quote pair ( c, q ) , the pseudo-rank score of q is p  = e q  c e q  c + (cid:80) q   N ( q ) e q   c , (5) where N ( q ) is the set of quotes selected as negative samples. The problem about quote encoder training has been largely solved, but the context encoder may be under-trained. The context encoder needs to process lots of contexts and thus requires more training than the quote encoder. Therefore, we adopt a two-stage training strategy. After the simultaneous training of quote and context encoders in the first stage, we continue to train the context encoder while freezing the quote encoder in the second stage. The training loss of the second stage is the cross-entropy loss among all quotes. Most quotes are quite pithy, and thus it is usually hard to learn their representations well. To obtain better quote representations, previous work tries incorporating external information, including the topic and author information of quotes, in the quote encoder (Tan et al., 2016, 2018). Although helpful, this external information is not always available or accurate  quite a few quotes are anonymous, and the topics attributed to quotes are usually from crowdsourcing and uninspected. We propose to incorporate sememe knowledge into quote representation learning, which is more general (every word can be annotated with sememes) and credible (the sememe annotations of words are given by experts). A sememe is the minimum semantic unit of human languages (Bloom-field, 1926), and it is believed that meanings of all words can be represented by a limited set of sememes. Sememe knowledge bases like HowNet (Dong and Dong, 2006) use a set of predefined sememes to annotate words, so that the meaning of a word can be precisely expressed by its sememes. With the help of such sememe knowledge bases, sememes have been successfully utilized in various NLP tasks (Qi et al., 2021a), including semantic composition (Qi et al., 2019), word sense disambiguation (Hou et al., 2020), reverse dictionary (Zhang et al., 2020a), adversarial attacks (Zang et al., 2020), backdoor learning (Qi et al., 2021b), etc. Inspired by the studies on incorporating sememes into recurrent neural networks (Qin et al., 2020) and transformers (Zhang et al., 2020b) to improve their representation learning ability, we adopt a similar way to incorporate sememes into the quote encoder. We simply add the average embedding of a word's sememes to every token embedding of the word in BERT. Formally, for a word in a quote that is divided into n tokens after tokenization w = x 1 ,    , x n , the embedding of its each token x i is transformed into x i  x i +  | S ( w ) | (cid:88) s j  S ( w ) s j ,  i = 1 ,    , n (6) where S ( w ) is the sememe set of the word w , and  is a hyper-parameter controlling the weight of sememe embeddings. Following previous work (Qin et al., 2020), the sememe embeddings are randomly initialized and updated during training. In this section, we evaluate our model and previous quote recommendation methods on QuoteR. We have three groups of approaches for comparison. The first group consists of two methods that widely serve as baselines in previous studies. (1.1) CRM , namely context-aware relevance model (He et al., 2010) that recommends the quote whose known contexts are most similar to the query context. (1.2) LSTM , which uses two LSTM encoders to learn representations of quotes and contexts. The second group includes representative approaches proposed in previous studies. (2.1) top-k RM , namely top-k rank multiplication (Ahn et al., 2016), which is a rank aggregation method based 341 Part English Standard Chinese Classical Chinese Model MRR NDCG  R /  R /  R Recall@1/10/100 MRR NDCG  R /  R /  R Recall@1/10/100 MRR NDCG  R /  R /  R Recall@1/10/100 CRM 0.192 0.193 599/1169/1408 16.51/23.66/32.78 0.397 0.407 13/325/584 33.60/49.32/61.70 0.198 0.203 166/548/811 14.52/28.79/44.51 LSTM 0.321 0.320 30/334/727 27.23/40.78/62.47 0.292 0.290 48/338/574 24.78/37.71/58.06 0.247 0.245 56/341/633 20.08/33.23/56.96 top-k RM 0.422 0.431 6/548/1243 35.99/53.31/66.20 0.480 0.494 3/377/774 40.17/60.67/72.26 0.294 0.299 48/511/980 23.54/39.58/56.90 NNQR 0.318 0.319 31/359/773 26.78/41.10/61.29 0.271 0.271 54/348/595 22.94/35.72/57.18 0.272 0.270 41/310/620 22.03/36.59/60.63 N-QRM 0.365 0.368 28/777/1465 32.24/44.41/58.26 0.343 0.347 55/575/890 30.20/41.22/54.15 0.287 0.288 98/917/1373 24.88/35.02/49.49 Transform 0.561 0.568 1/241/749 50.11/65.88/79.98 0.512 0.519 2/271/576 45.50/60.31/72.83 0.449 0.453 5/269/663 39.01/55.78/73.58 BERT-Sim 0.526 0.529 2/487/1064 49.38/58.05/67.75 0.500 0.508 2/229/511 44.47/59.07/72.21 0.439 0.443 7/320/711 38.85/53.04/68.32 BERT-Cls 0.310 0.329 7/134/453 18.15/57.11/82.05 0.378 0.395 5/152/413 26.88/57.90/78.38 0.330 0.345 8/ 135 / 377 21.93/54.27/78.75 Ours 0.572 0.580 1/ 123 / 433 50.74/ 69.03 / 83.84 0.541 0.548 2/ 139 / 370 47.91 / 64.97 / 79.35 0.484 0.490 3/146/422 41.67 / 60.78 / 79.38 -Sememe 0.568 0.574 1/145/492 51.05 /67.07/82.34 0.535 0.543 2/160/402 47.62/63.66/77.68 0.475 0.481 3/152/435 40.93/60.26/78.39 -ReTrain 0.299 0.307 12/176/503 20.46/47.89/75.74 0.255 0.260 20/210/435 16.87/42.94/68.43 0.265 0.269 17/184/450 17.87/43.56/72.89 -SimTrain 0.529 0.532 2/467/1060 49.31/58.97/69.48 0.519 0.526 2/204/489 46.00/62.03/75.34 0.465 0.470 4/310/713 41.40/55.53/70.09 Table 2: Quote recommendation results on the three parts of QuoteR. on the ensemble of a statistical method, random forest, CNN and LSTM. (2.2) NNQR (Tan et al., 2016), which reforms LSTM by incorporating additional quote information (topic and author) into the quote encoder and perturbing the word embeddings of quotes. (2.3) N-QRM (Tan et al., 2018), which further improves NNQR mostly by adjusting the training loss to prevent overfitting. (2.4) Transform (Wang et al., 2021), which uses Trans-former+GRU to encode contexts and transforms context embeddings into the space of quote embeddings learned from another Transformer. 10 The third group comprises two BERT-based approaches that are frequently utilized in sentence matching and sentence pair classification. (3.1) BERT-Sim , which is the vanilla BERT-based model discussed in 5.1. It directly uses the hidden states of the [CLS] tokens as the representations of both quotes and contexts, and freezes the quote encoder during training, as explained in 5.2. (3.2) BERT-Cls , which conducts a binary classification for the concatenation of the query context and a candidate quote. Following previous work (Ahn et al., 2016; Tan et al., 2018), we use three evaluation metrics: (1) Mean reciprocal rank ( MRR ), the average reciprocal values of the ranks of the gold quotes; (2) Normalized discounted cumulative gain ( NDCG@K ) (Jrvelin and Keklinen, 2002), a widely used measure of ranking quality and is computed by where r ( i ) = 1 if the i -th quote is the gold quote, otherwise r ( i ) = 0 , ZK = 1 is a normalization constant. We report the average of NDCG@5 scores of all the evaluated query contexts. (3) Re-call@K , the proportion of query contexts whose gold quotes are ranked in respective top K candidate quotes, K = { 1 , 10 , 100 } . Besides, we use another three evaluation metrics: (4) Median Rank (  R ), (5) Mean Rank (  R ) and (6) Rank Variance (  R ), the median, average and standard deviation of the ranks of gold quotes. The higher MRR, NDCG@K and Recall@K and the lower  R ,  R and  R are, the better a model is. 6.3 Implementation Details We use BERTBASE for both English and Chinese from Transformers (Wolf et al., 2020). We use the AdamW optimizer (Loshchilov and Hutter, 2018) with an initial learning rate 5e-5 that gradually declines to train our model. We randomly select N negative samples, and N is tuned in {4,9,19,29,39} on the validation set. The weight of sememe embeddings  is tuned in {0.1, 0.2, 0.5, 1.0, 2.0}. The underlined numbers are final picks. For the previous methods, we use their original hyperparameters and experimental settings given in the papers. Table 2 lists the evaluation results of different methods on the three parts of QuoteR. We observe that (1) our method achieves the best overall results and displays its superiority to other methods; (2) the two BERT-based models, especially BERT-Sim, yield quite high performance, which reflects the importance of a powerful sentence encoder to quote recommendation; (3) among the three parts, almost all methods perform worse on Classical Chinese, 342 Part English Standard Chinese Classical Chinese Model MRR NDCG  R /  R /  R Recall@1/10/100 MRR NDCG  R /  R /  R Recall@1/10/100 MRR NDCG  R /  R /  R Recall@1/10/100 CRM 0.154 0.156 353/948/1297 11.88/21.78/33.66 0.292 0.296 124/401/524 25.28/35.39/48.43 0.141 0.146 276/587/763 9.88/19.75/34.57 LSTM 0.272 0.271 89/552/992 23.38/33.87/51.12 0.210 0.208 146/483/662 18.26/27.67/45.50 0.182 0.178 117/465/750 13.87/25.44/47.80 top-k RM 0.360 0.366 30/833/1497 31.20/44.55/56.80 0.350 0.358 38/620/926 29.77/44.40/55.53 0.276 0.280 77/645/1088 22.61/36.16/52.57 NNQR 0.267 0.266 98/592/1043 22.82/33.48/50.28 0.224 0.223 145/495/683 17.16/27.67/45.81 0.189 0.187 98/441/766 14.18/26.86/50.29 N-QRM 0.270 0.272 156/1145/1735 23.40/33.18/46.54 0.266 0.270 287/778/946 21.27/30.63/42.32 0.215 0.215 356/1232/1505 17.72/27.13/40.73 Transform 0.438 0.443 6/429/1036 38.47/53.43/68.65 0.371 0.374 29/465/748 32.54/44.83/58.04 0.331 0.334 29/435/842 27.76/42.87/60.85 BERT-Sim 0.399 0.401 44/839/1407 36.95/44.75/54.32 0.364 0.370 41/431/695 31.71/44.28/56.18 0.310 0.313 56/522/902 26.32/39.05/54.56 BERT-Cls 0.265 0.275 15/ 237 / 640 16.75/45.37/71.77 0.213 0.220 24/318/646 12.47/40.53/64.67 0.204 0.208 25/253/568 11.50/38.27/66.73 Ours 0.456 0.462 4 /254/685 39.62 / 56.21 / 73.26 0.413 0.419 7 / 97 / 186 34.64 / 53.29 / 75.91 0.409 0.411 9 / 196 / 419 35.22 / 51.47 / 70.82 Table 3: Quote recommendation results on the three parts of QuoteR, given the left context only . which is presumably because Chinese BERT is pre-trained on standard Chinese corpora and not suitable to encode the classical Chinese quotes. We conduct ablation studies to investigate the effectiveness of our training strategy and the incorporation of sememes. We first remove the incorporation of sememes (-Sememe), then further do not separately train the context encoder after the simultaneous training of the context and quote encoders (-ReTrain), and finally discard the simultaneous training of the two encoders and train the context encoder only (-SimTrain). -SimTrain differs BERT-Sim only in the choice of context representation ( [MASK] vs. [CLS] ). The results of ablation studies are given in the last three rows of Table 2. We have the following observations: (1) -Sememe causes consistent performance decline as compared to Ours, which demonstrates the role of sememes in improving quote encoding, thereby benefiting quote recommendation; (2) the performance of -ReTrain is pretty poor, which reflects the necessity of separate training for the context encoder after simultaneous training; (3) -SimTrain is inferior to -Sememe, which displays the usefulness of simultaneously training the two encoders; (4) -SimTrain outperforms BERT-Sim, proving the superiority of choosing [MASK] to represent contexts in our method. Following previous work (Tan et al., 2015; Ahn et al., 2016; Tan et al., 2018), the evaluation experiments are mainly conducted in the setting where both the left and right contexts are given. However, in practical terms, quote recommendation given the left context only might be more useful. Therefore, we also conduct experiments in the setting where only the left context is given. Table 3 shows the 0 1 (1,5] (5,10] (10,20] (20,50] (50,100] (100,150] Number of Contexts in the Training Set 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 MRR / NDCG @ 5 MRR NDCG@5 Figure 2: Recommendation performance for quotes within different occurrence frequency ranges. results. We can see that our method is still the best one on all three parts. In addition, the performance of all methods decreases substantially, which indicates that both the left and right contexts provide important information for quote recommendation. In this subsection, we investigate the effect of the gold quote's occurrence frequency on recommendation performance. Figure 2 shows MRR and NDCG@5 results for quotes that have different numbers of contexts in the training set of the standard Chinese part. We observe that the occurrence frequency has great impact on quote recommendation performance. Basically, increasing occurrences of quotes in the training set can increase recommendation performance, because we can learn better representations for the quotes with more adequate training. But the most frequent quotes does not have the best performance, possibly because these quotes carry very rich semantics and can be cited in various contexts , which makes it very hard to correctly recommend them. In addition, the performance for the unseen quotes is very limited. It reflects the 343 #NS MRR NDCG  R /  R /  R Recall@1/10/100 4 0.533 0.540 2 / 161 / 412 47.48 / 63.23 / 77.68 9 0.534 0.541 2 / 148 / 381 47.50 / 63.97 / 78.83 19 0.541 0.548 2 / 139 / 370 47.91 / 64.97 / 79.35 29 0.545 0.552 2 / 174 / 434 47.06 / 63.58 / 76.92 39 0.535 0.543 2 / 132 / 357 47.17 / 64.97 / 79.43 Table 4: Quote recommendation results with different negative sample numbers (#NS). In this subsection, we investigate the effect of the negative sample number (#NS), a hyper-parameter of our method, on quote recommendation performance. Table 4 gives the results of different negative sample numbers on the validation set of the standard Chinese part of QuoteR. We can see that increasing negative samples (from 4 to 19) can increase quote recommendation performance, which is because the quote encoder can be trained more sufficiently. However, when the negative samples continue increasing, the performance fluctuates or even decreases. That is possibly because of the imbalance of positive and negative samples (there is only one positive sample, namely the gold quote), as explained in 5.2. Therefore, taking both performance and computation efficiency into consideration, we choose 19 as the final negative sample number. As mentioned in 3, there may be other quotes that are suitable for a query context besides the gold quote. Hence, we conduct a human evaluation on the recommendation results of our method. We randomly select 50 contexts from the validation set of the standard Chinese part and list the top 10 quotes recommended by our method for each context. Then we ask annotators to make a binary suitability decision for the quotes. Each quote is annotated by 3 native speakers and the final decision is made by voting. For each context, we regard the suitable quote with the highest ranking as the gold quote, and re-evaluate the recommendation performance: NDCG@5=0.661, Recall@1/10=0.50/0.92. 11 In contrast, the original evaluation results among the 50 contexts are 11 Since we only annotate the top 10 results, there are no other available metrics than NDCG@5 and Recall@1/10. NDCG@5=0.439 , Recall@1/10=0.36/0.64. By comparison, we can conclude that the real performance of our method is substantially underestimated. We also count the average number of suitable quotes among the top 10 quotes, which is 1.76. We feed the context in Figure 1 into our model, and print the top 5 recommended quotes and their rank scores in Table 5. We find that the gold quote is ranked second, but the first one is actually another statement version of the gold quote and has exactly the same meaning. In addition, the third and fourth quotes are also related to the context. This case, together with more cases in Appendix B, can demonstrate the practical effectiveness and usefulness of our model. In this paper, we build a large and the first open dataset of quote recommendation for writing named QuoteR and conduct an extensive evaluation of existing quote recommendation methods on it. We also propose a new model that achieves absolute outperformance over previous methods, and its effectiveness is proved by ablation studies. In the future, we will try to improve our model in handling classical Chinese quotes by using a special classical Chinese pre-trained model to encode them. We will also consider boosting the performance of our model in the few-shot and zero-shot situations. This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute Guo Qiang at Tsinghua University, and International Innovation Center of Tsinghua University, Shanghai, China. We also thank all the anonymous reviewers for their valuable comments and suggestions. Dataset and Human Evaluation In terms of our QuoteR dataset, all the quotes are collected from free and open quote repository websites. Besides, all the contexts are extracted from open corpora, including free public domain e-books and other open corpora. Therefore, there is no intellectual property problem for the dataset. In addition, we conduct the human evaluation by a reputable data annotation company. The annotators are fairly compensated by the company, based on the previous annotation tasks. Further, we do not directly communicate with the annotators, so that their privacy is well preserved. Finally, the dataset and the human evaluation are not sensitive and thus do not need to be approved by the institutional review board (IRB). Application Quote recommendation is a practical task and our model can be put into service. In actual use cases, users just need to input a query context and our model should output a list of candidate quotes that fit the given context. All people may benefit from our model during writing. If our model fails, some inappropriate quotes that cannot fit the query context would be output, but no one would be harmed. There are indeed biases in the dataset we build. Some quotes are very frequent while the others are not, as illustrated in 6.6. The infrequent quotes are less recommended and may cause the failure of our model in some cases. In terms of misuse, to the best of our knowledge, such a quote recommendation model is hardly misused. After the deployment of our model, the system would not collect data from users. It does not have any potential harm to vulnerable populations, either. Energy Saving To save energy, we use the base version of BERT rather than larger pre-trained language models, although the larger ones would probably yield better performance. Besides, as discussed in 5.2, we find that the simultaneous training of the context and quote encoders requires very big memory and computation resources, and thus we adopt the strategy of negative sampling in training. Use of Identity Characteristics In this work, we do not use any demographic or identity characteristics information.
method abstain abstain method abstain method objective other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective other abstain method objective abstain abstain method abstain method abstain abstain objective abstain abstain objective abstain abstain objective other other other other other other other other other other other other other other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain method abstain method abstain method method method result abstain abstain method abstain abstain abstain method method result method abstain method abstain result method result abstain abstain other result abstain method method method result result abstain method method method result abstain abstain method other abstain abstain abstain abstain abstain method abstain other method abstain abstain method method abstain other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain other abstain method other abstain abstain method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain other abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective result method other other method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain method abstain method