We present a content-based method for recommending citations in an academic paper draft. We embed a given query document into a vector space, then use its nearest neighbors as candidates, and rerank the candidates using a discriminative model trained to distinguish between observed and unobserved citations. Unlike previous work, our method does not require metadata such as author names which can be missing, e.g., during the peer review process. Without using metadata, our method outperforms the best reported results on PubMed and DBLP datasets with relative improvements of over 18% in F1@20 and over 22% in MRR. We show empirically that, although adding metadata improves the performance on standard metrics, it favors self-citations which are less useful in a citation recommendation setup. We release an online portal for citation recommendation based on our method, 1 and a new dataset OpenCorpus of 7 million research articles to facilitate future research on this task. Due to the rapid growth of the scientific literature, conducting a comprehensive literature review has become challenging, despite major advances in digital libraries and information retrieval systems. Citation recommendation can help improve the quality and efficiency of this process by suggesting published scientific documents as likely citations for a query document, e.g., a paper draft to be submitted for ACL 2018. Existing citation recommendation systems rely on various information of the query documents such as author names and publication venue (Ren et al., 2014; Yu et al.,  Work done while on contract with AI2  Work done while at AI2 1 http://labs.semanticscholar.org/citeomatic/ 2012), or a partial list of citations provided by the author (McNee et al., 2002; Liu et al., 2015; Jia and Saule, 2017) which may not be available, e.g., during the peer review process or in the early stage of a research project. Our method uses a neural model to embed all available documents into a vector space by encoding the textual content of each document. We then select the nearest neighbors of a query document as candidates and rerank the candidates using a second model trained to discriminate between observed and unobserved citations. Unlike previous work, we can embed new documents in the same vector space used to identify candidate citations based on their text content, obviating the need to re-train the models to include new published papers. Further, unlike prior work (Yang et al., 2015; Ren et al., 2014), our model is computationally ef-ficient and scalable during both training and test time. We assess the feasibility of recommending citations when some metadata for the query document is missing, and find that we are able to outperform the best reported results on two datasets while only using papers' textual content (i.e. its title and ab-stract). While adding metadata helps further improve the performance of our method on standard metrics, we found that it introduces a bias for self-citation which might not be desirable in a citation recommendation system. See  5 for details of our experimental results. Our main contributions are:  a content-based method for citation recommendation which remains robust when metadata are missing for query documents,  large improvements over state of the art results on two citation recommendation datasets despite omitting the metadata,  a new dataset of seven million research papers, addressing some of the limitations in 238 d q d 1 d 2 d 3 d 4 d 5 d 6 d 7 d q d 1 d 2 d 3 d 6 d 7 d 4 d 5 d q d 1 cites d 4 d 1 cites d 5 d 3 cites d 7  K=4 Phase 1: candidate selection Phase 2: reranking NNRank d 2 0.7 rerankedlist cited in nearest neighbors: d q NNRank d 6 0.8 d q NNRank d 3 0.5 d q NNRank d 4 0.3 d q NNRank d 7 0.9 d 7 d 6 d 2 d 3 d 4 t o p N = 3 r e c o mm e nd a t i o n s query document document embeddings nearest neighbors of d q : Figure 1: An overview of our Citation Recommendation system. previous datasets used for citation recommendation, and  a scalable web-based literature review tool based on this work. 2 2 Overview We formulate citation recommendation as a ranking problem. Given a query document d q and a large corpus of published documents, the task is to rank documents which should be referenced in d q higher than other documents. Following previous work on citation recommendation, we use standard metrics (precision, recall, F-measure and mean reciprocal rank) to evaluate our predictions against gold references provided by the authors of query documents. Since the number of published documents in the corpus can be large, it is computationally expensive to score each document as a candidate reference with respect to d q . Instead, we recommend citations in two phases: (i) a fast, recall-oriented candidate selection phase, and (ii) a feature rich, 2 https://github.com/allenai/citeomatic precision-oriented reranking phase. Figure 1 provides an overview of the two phases using a toy example. Phase 1 Candidate Selection: In this phase, our goal is to identify a set of candidate references for d q for further analysis without explicitly iterating over all documents in the corpus. 3 Using a trained neural network, we first project all published documents into a vector space such that a document tends to be close to its references. Since the projection of a document is independent of the query document, the entire corpus needs to be embedded only once and can be reused for subsequent queries. Then, we project each query document d q to the same vector space and identify its nearest neighbors as candidate references. See  3 for more details about candidate selection. 3 In order to increase the chances that all references are present in the list of candidates, the number of candidates must be significantly larger than the total number of citations of a document, but also significantly smaller than the number of documents in the corpus. Phase 2 Reranking: Phase 1 yields a manageable number of candidates making it feasible to score each candidate d i by feeding the pair ( d q , d i ) into another neural network trained to discriminate between observed and unobserved citation pairs. The candidate documents are sorted by their estimated probability of being cited in d q , and top candidates are returned as recommended citations. See  4 for more details about the reranking model and inference in the candidate selection phase. In this phase, we select a pool of candidate citations for a given query document to be reranked in the next phase. First, we compute a dense embedding of the query document d q using the document embedding model (described next), and select K nearest neighbor documents in the vector space as candidates. 4 Following Strohman et al. (2007), we also include the outgoing citations of the K nearest neighbors as candidates. The output of this phase is a list of candidate documents d i and their corresponding scores NNSelect ( d q , d i ) , defined as the cosine similarity between d q and d i in the document embedding space. Document embedding model. We use a supervised neural model to project any document d to a dense embedding based on its textual content. We use a bag-of-word representation of each textual field, e.g., d [ title ] = { content-based', citation', recommendation' } , and compute the feature vector: f d [ title ] = X t  d [ title ] w mag t w dir t k w dir t k 2 , (1) where w dir t is a dense direction embedding and w mag t is a scalar magnitude for word type t . 5 We then normalize the representation of each field and compute a weighted average of fields to get the document embedding, e d . In our experiments, we use the title and abstract fields of a document d : e d =  title f d [ title ] k f d [ title ] k 2 +  abstract f d [ abstract ] k f d [ abstract ] k 2 , 4 We tune K as a hyperparameter of our method. 5 The magnitude-direction representation is based on Salimans and Kingma (2016) and was found to improve results in preliminary experiments, compared to the standard direction-only word representation. Training. We learn the parameters of the document embedding model (i.e.,   , w mag  , w dir  ) using a training set T of triplets h d q , d + , d  i where d q is a query document, d + is a document cited in d q , and d  is a document not cited in d q . The model is trained to predict a high cosine similarity for the pair ( d q , d + ) and a low cosine similarity for the pair ( d q , d  ) using the per-instance triplet loss (Wang et al., 2014): loss = max (cid:0)  + s ( d q , d  )  s ( d q , d + ) , 0 (cid:1) , (2) where s ( d i , d j ) is defined as the cosine similarity between document embeddings cos-sim ( e d i , e d j ) . We tune the margin  as a hyperparameter of the model (see Appendix B for more details). Next, we describe how negative examples are selected. Selecting negative examples. Defining positive examples is straight-forward; we use any ( d q , d + ) pair where a document d q in the training set cites d + . However, a careful choice of negative training examples is critical for model performance. We use three types of negative examples: 1. Random: any document not cited by d q . 2. Negative nearest neighbors: documents that are close to d q in the embedding space, but are not cited in it. 6 3. Citation-of-citation: documents referenced in positive citations of d q , but are not cited directly in d q . Negative examples belong to at least one of these types that serve different, and complementary purposes. Selecting a paper from the corpus at random as a negative example typically results in easy negative examples. Selecting nearest neighbor documents in the embedding space used for candidate selection enables the re-ranking phase (described in  4) to fix some of the mistakes made in the candidate selection step. Finally, using citations-of-citations as negative examples is based on the assumption that the authors would have included them as positive examples if they were relevant for the query paper. In Appendix  A, we describe the number of negative examples of each type used for training. Next, we describe how to rerank the candidate documents. Since the set of approximate neighbors depend on model parameters, we recompute a map from each query document to its K nearest neighbors before each epoch while training the document embedding model. In this phase, we train another model which takes as input a pair of documents ( d q , d i ) and estimates the probability that d i should be cited in d q . Input features. A key point of this work is to assess the feasibility of recommending citations without using metadata, but we describe all features here for completeness and defer this discussion to  5. For each document, we compute dense feature vectors f d [ field ] as defined in Eq. 1 for the following fields: title, abstract, authors, venue and keyphrases (if available). For the title and abstract, we identify the subset of word types which appear in both documents (intersection), and compute the sum of their scalar weights as an additional feature, e.g., P t  title w  t . We also use log number of times the candidate document d i has been cited in the corpus, i.e., log ( d i [ in-citations ] ). Finally, we use the cosine similarity between d q and d i in the embedding space, i.e., cos-sim ( e d q , e d i ) . output layer is defined as: s ( d i , d j ) = FeedForward ( h ) , (3) h = h g title ; g abstract ; g authors ; g venue ; g keyphrases ; cos-sim ( e d q , e d i ); P t  title w  t ; P t  abstract w  t ; d i [ in-citations ] i , g field = cos-sim ( f d [ field ] , f d [ field ] ) , where FeedForward' is a three layer feed-forward neural network with two exponential linear unit layers (Clevert et al., 2015) and one sigmoid layer. ';' indicates concatenation. Training. The parameters of the NNRank model are w mag  , w dir  , w  and parameters of the three dense layers in FeedForward'. We reuse the triplet loss in Eq. 2 to learn these parameters, but redefine the similarity function s ( d i , d j ) as the sigmoid output described in Eq. 3. At test time, we use this model to recommend candidates d i with the highest s ( d q , d i ) scores. In this section, we describe experimental results of our citation recommendation method and compare it to previous work. Datasets. We use the DBLP and PubMed datasets (Ren et al., 2014) to compare with previous work on citation recommendation. The DBLP 241 dataset contains over 50K scientific articles in the computer science domain, with an average of 5 citations per article. The PubMed dataset contains over 45K scientific articles in the medical domains, with an average of 17 citations per article. In both datasets, a document is accompanied by its title, abstract, venue (i.e. journal or conference where the document was published), authors, citations (i.e. other documents in the corpus that are referenced in the given document) and keyphrases (i.e. phrases considered important by automated extraction methods). We replicate the experimental setup of Ren et al. (2014) by excluding papers with fewer than 10 citations and using the standard train, dev and test splits. 7 We also introduce OpenCorpus , 8 a new dataset of 7 million scientific articles primarily drawn from the computer science and neuroscience domain. Due to licensing constraints, documents in the corpus do not include the full text of the scientific articles, but include the title, abstract, year, author, venue, keyphrases and citation information. The mutually exclusive training, development, and test splits were selected such that no document in the development or test set has a publication year less than that of any document in the training set. Papers with zero citations were removed from the development and test sets. We describe the key characteristics of OpenCorpus in Table 1. Statistic Value # of documents in corpus 6.9 million # of unique authors 8.3 million # of unique keyphrases 823,677 # of unique venues 23,672 avg. Baselines. We compare our method to two baseline methods for recommending citations: ClusCite and BM25. ClusCite (Ren et al., 2014) clusters nodes in a heterogeneous graph of terms, authors and venues in order to find related documents which should be cited. We use the ClusCite results 7 The dataset characteristics reported here are different from those in Table 3 in (Ren et al., 2014) because we report the size of the filtered datasets while they report the size of the datasets before filtering. as reported in Ren et al. (2014), which compared it to several other citation recommendation methods and found that it obtains state of the art results on the PubMed and DBLP datasets. The BM25 results are based on our implementation of the popular ranking function Okapi BM25 used in many information retrieval systems. See Appendix  D for details of our BM25 implementation. Evaluation. We use Mean Reciprocal Rank (MRR) and F1@20 to report the main results in this section. In Appendix  F, we also report additional metrics (e.g., precision and recall at 20) which have been used in previous work. We compute F1@20 as the harmonic mean of the corpus-level precision and recall at 20 (P@20 and R@20). Following (Ren et al., 2014), precision and recall at 20 are first computed for each query document then averaged over query documents in the test set to compute the corpus-level P@20 and R@20. Configurations. To find candidates in NNSelect , we use the approximate nearest neighbor search algorithm Annoy 9 , which builds a binary-tree structure that enables searching for nearest neighbors in O (log n ) time. To build this tree, points in a high-dimensional space are split by choosing random hyperplanes. We use 100 trees in our approximate nearest neighbors index, and retrieve documents using the cosine distance metric. We use the hyperopt library 10 to optimize various hyperparameters of our method such as size of hidden layers, regularization strength and learning rate. To ensure reproducibility, we provide a detailed description of the parameters used in both NNSelect and NNRank models, our hyperparameter optimization method and parameter values chosen in Appendix  A. Main results. Table 2 reports the F1@20 and MRR results for the two baselines and three variants of our method. Since the OpenCorpus dataset is much bigger, we were not able to train the ClusCite baseline for it. Totti et al. (2016) have also found it difficult to scale up ClusCite to larger datasets. Where available, we report the mean  standard deviation based on five trials. The first variant, labeled  NNSelect , only uses the candidate selection part of our method (i.e., phase 1) to rank candidates by their cosine 9 https://github.com/spotify/annoy 10 https://github.com/hyperopt/hyperopt 242 Method DBLP PubMed OpenCorpus F1@20 MRR F1@20 MRR F1@20 MRR BM25 0.119 0.425 0.209 0.574 0.058 0.218 ClusCite 0.237 0.548 0.274 0.578   NNSelect 0.282  0.002 0.579  0.007 0.309  0.001 0.699  0.001 0.109 0.221 + NNRank 0.302  0.001 0.672  0.015 0.325  0.001 0.754  0.003 0.126 0.330 + metadata 0.303  0.001 0.689  0.011 0.329  0.001 0.771  0.003 0.125 0.330 Table 2: F1@20 and MRR results for two baselines and three variants of our method. similarity to the query document in the embedding space as illustrated in Fig. 1. Although the document embedding space was designed to efficiently select candidates for further processing in phase 2, recommending citations directly based on the cosine distance in this space outperforms both baselines. The second variant, labeled  NNSelect + NNRank , uses the discriminative model (i.e., phase 2) to rerank candidates selected by NNSelect , without encoding metadata (venues, authors, keyphrases). Both the first and second variants show that improved modeling of paper text can significantly outperform previous methods for citation recommendation, without using metadata. The third variant, labeled  NNSelect + NNRank + metadata, further encodes the metadata features in the reranking model, and gives the best overall results. On both the DBLP and PubMed datasets, we obtain relative improvements over 20% (for F1@20) and 25% (for MRR) compared to the best reported results of ClusCite. In the rest of this section, we describe controlled experiments aimed at analyzing different aspects of our proposed method. Choice of negative samples. As discussed in  3, we use different types of negative samples to train our models. We experimented with using only a subset of the types, while controlling for the total number of negative samples used, and found that using negative nearest neighbors while training the models is particularly important for the method to work. As illustrated in Table 3, on the PubMed dataset, adding negative nearest neighbors while training the models improves the F1@20 score from 0.306 to 0.329, and improves the MRR score from 0.705 to 0.771. Intuitively, using nearest neighbor negative examples focuses training on the harder cases on which the model is more likely to make mistakes. Valuable features. We experimented with different subsets of the optional features used in NNRank in order to evaluate the contribution of various features. We found intersection features, NNSelect scores, and the number of incoming citations to be the most valuable feature. As illustrated in Table 3, the intersection features improves the F1@20 score from 0.296 to 0.329, and the MRR score from 0.653 to 0.771, on the PubMed dataset. The numerical features ( NNSelect score and incoming citations) improve the F1@20 score from 0.314 to 0.329, and improves the MRR score from 0.735 to 0.771. This shows that, in some applications, feeding engineered features to neural networks can be an effective strategy to improve their performance. Performance across venues We studied the variability of performance of our model for papers from different venues. Figure 3 shows the F1@20 score of NNRank for papers belonging to the top 243 0 0 . ten venues (by their paper count) in the Pubmed corpus. NNRank 's performance is robust across venues. Encoding textual features. We also experimented with using recurrent and convolutional neural network to encode the textual fields of query and candidate documents, instead of using a weighted sum as described in Eq. 1. We found that recurrent and convolutional encoders are much slower, and did not observe a significant improvement in the overall performance as measured by the F1@20 and MRR metrics. This result is consistent with previous studies on other tasks, e.g., Iyyer et al. (2015). Number of nearest neighbors. As discussed in  3, the candidate selection step is crucial for the scalability of our method because it reduces the number of computationally expensive pairwise comparisons with the query document at runtime. We did a controlled experiment on the OpenCorpus dataset (largest among the three datasets) to measure the effect of using different numbers of nearest neighbors, and found that both P@20 and R@20 metrics are maximized when NNSelect fetches five nearest neighbors using the approximate nearest neighbors index (and their out-going citations), as illustrated in Table 4. Self-citation bias. We hypothesized that a model trained with the metadata (e.g., authors) could be biased towards self-citations and other well-cited authors. To verify this hypothesis, we compared two NNRank models  one with meta-# of neighbors R@20 P@20 Time(ms) 1 0.123 0.079 131 5 0.142 0.080 144 10 0.138 0.069 200 50 0.081 0.040 362 Table 4: OpenCorpus results for NNSelect step with varying number of nearest neighbors on 1,000 validation documents. data, and one without. We measured the mean and max rank of predictions that had at least one author in common with the query document. This experiment was performed with the OpenCorpus dataset. A lower mean rank for NNRank + Metadata indicates that the model trained with metadata tends to favor documents authored by one of the query document's authors. We verified the prevalence of this bias by varying the number of predictions for each model from 1 to 100. Figure 4 shows that the mean and max rank of the model trained with metadata is always lower than those for the model that does not use metadata. Citation recommendation systems can be divided into two categories  local and global . A local citation recommendation system takes a few sentences (and an optional placeholder for the candidate citation) as input and recommends citations based on the local context of the input sen-244 tences (Huang et al., 2015; He et al., 2010; Tang and Zhang, 2009; Huang et al., 2012; He et al., 2011). A global citation recommendation system takes the entire scholarly article as input and recommends citations for the paper (McNee et al., 2002; Strohman et al., 2007; Nallapati et al., 2008; Kataria et al., 2010; Ren et al., 2014). We address the global citation recommendation problem in this paper. A key difference of our proposed method compared to previous work is that our method is content-based and works well even in the absence of metadata (e.g. authors, venues, key phrases, seed list of citations). Many citation recommendation systems crucially rely on a query document's metadata. For example, the collaborative filtering based algorithms of McNee et al. (2002); Jia and Saule (2017); Liu et al. (2015) require seed citations for a query document. (Ren et al., 2014; Yu et al., 2012) require authors, venues and key terms of the query documents to infer interest groups and to extract features based on paths in a heterogeneous graph. In contrast, our model performs well solely based on the textual content of the query document. Some previous work (e.g. (Ren et al., 2014; Yu et al., 2012)) have addressed the citation recommendation problem using graph-based methods. But, training graph-based citation recommendation models has been found to be expensive. For example, the training complexity of the ClusCite algorithm (Ren et al., 2014) is cubic in the number of edges in the graph of authors, venues and terms. This can be prohibitively expensive for datasets as large as OpenCorpus . On the other hand our model is a neural network trained via batched stochastic gradient descent that scales very well to large datasets (Bottou, 2010). Another crucial difference between our approach and some prior work in citation prediction is that we build up a document representation using its constituent words only. Prior algorithms (Huang et al., 2015, 2012; Nallapati et al., 2008; Tanner and Charniak, 2015) learn an explicit representation for each training document separately that isn't a deterministic function of the document's words. This makes the model effectively transductive since a never-before-seen document does not have a ready-made representation. Similarly, Huang et al. (2012)'s method needs a candidate document to have at least one in-coming citation to be eligible for citation  this disadvantages newly published documents. Liu et al. (2015) form document representations using citation relations, which are not available for unfinished or new documents. In contrast, our method does not need to be re-trained as the corpus of potential candidates grows. As long as the new documents are in the same domain as that of the model's training documents, they can simply be added to the corpus and are immediately available as candidates for future queries. While the citation recommendation task has attracted a lot of research interest, a recent survey paper (Beel et al., 2016) has found three main concerns with existing work: (i) limitations in evaluation due to strongly pruned datasets, (ii) lack of details for re-implementation, and (iii) variations in performance across datasets. For example, the average number of citations per document in the DBLP dataset is 5, but Ren et al. (2014) filtered out documents with fewer than 10 citations from the test set. This drastically reduced the size of the test set. We address these concerns by releasing a new large scale dataset for future citation recommendation systems. In our experiments on the OpenCorpus dataset, we only prune documents with zero outgoing citations. We provide extensive details of our system (see Appendix  A) to facilitate reproducibility and release our code 11 . We also show in experiments that our method consistently outperforms previous systems on multiple datasets. Finally, recent work has combined graph node representations and text-based document representations using CCA (Gupta and Varma, 2017). This sort of approach can enhance our text-based document representations if a technique to create graph node representations at test-time is available. In this paper, we present a content-based citation recommendation method which remains robust when metadata is missing for query documents, enabling researchers to do an effective literature search early in their research cycle or during the peer review process, among other scenarios. We show that our method obtains state of the art results on two citation recommendation datasets, even without the use of metadata available to the 11 https://github.com/allenai/citeomatic 245 baseline method. We make our system publicly accessible online. We also introduce a new dataset of seven million scientific articles to facilitate future research on this problem. We would like to thank Oren Etzioni, Luke Zettle-moyer, Doug Downey and Iz Beltagy for participating in discussions and for providing helpful comments on the paper draft; Hsu Han and rest of the Semantic Scholar team at AI2 for creating the OpenCorpus dataset. We also thank Xiang Ren for providing the data used in their experiments on the DBLP and Pubmed datasets. Finally, we thank the anonymous reviewers for insightful comments on the draft.
method method abstain result result objective abstain abstain other method method objective abstain result result result other method method abstain method abstain method abstain other abstain objective objective abstain method abstain abstain abstain abstain abstain abstain objective method abstain abstain method method method method abstain abstain method other result abstain abstain method abstain method objective objective abstain abstain abstain abstain method abstain method method abstain method abstain method method method other abstain abstain abstain abstain abstain abstain method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain result abstain abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain other abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain other other other abstain objective other other other objective other other other other method method other other other other objective other other other other other other other objective abstain abstain abstain other method method other abstain objective other other other