Measuring the scholarly impact of a document without citations is an important and challenging problem. Existing approaches such as Document Influence Model (DIM) are based on dynamic topic models, which only consider the word frequency change. In this paper, we use both frequency changes and word semantic shifts to measure document influence by developing a neural network based framework. Our model has three steps. Firstly, we train word embeddings for different time periods. Subsequently, we propose an unsupervised method to align vectors for different time periods. Finally, we compute the influence value of documents. Our experimental results show that our model outperforms DIM. Identifying the most influential articles is of great importance in many areas of research. It is often the case that we are increasingly exposed to numerous papers published every day. Research on influence evaluation can be applied to measure the scholarly impact of universities and research facilities. Besides, it helps researchers to distinguish valuable research work from a large number of sci-entific papers. The common approach of assessing an article's research impact is to count the number of explicit references to it. However, citations are often not available. For example, collections including blog posts and government documents adopt ideas proposed in the documents without explicit references (Stringer et al., 2008; Macroberts and Macroberts, 2010). To identify influential articles without citations, Gerrish and Blei (2010) and Gerow et al. (2018) proposed probabilistic methods, which are based on dynamic topic models (Blei and Lafferty, 2006). Corresponding Author. They aimed to identify influential articles by examining the word frequency change over time. In this paper, we aim to use both word frequency changes and word semantic shifts on measuring document influence without citations. For our purpose, we propose a neural network based method called Neural-DINF, which stands for a Neural Network based Framework for measuring Document Influence. Our idea is that words that have semantic shifts across time contribute significantly to the influence of a document. Recent studies show that words whose word embeddings across different time periods diverge significantly are suspected to have semantic shifts (Kim et al., 2014; Kulkarni et al., 2015; Hamilton et al., 2016). Neural-DINF first generates static word embeddings in each time period by using Word2Vec (Mikolov et al., 2013b,a) independently, then aligns embeddings to the same vector space with an unsupervised method, subsequently calculates differences of the embeddings of many words across time to identify words that experience semantic shifts, finally measures the influence of a document by counting these crucial words. In summary, this paper makes the following main contributions:  We consider both word frequency changes and word semantic shifts on measuring document influence without citations by developing a novel neural network framework. In the semantic change detection step, we propose an unsupervised method to align word embeddings across time. Neural-DINF outperforms dynamic topic based models such as DIM, which only considers the word frequency change. This paper is organized as follows: Section 2 states related work; Section 3 formulates our approach; Section 4 presents our experiments; Section 5 concludes our work. There are two lines of literature that are closely related to our work: document influence evaluation and semantic shift detection. Assessing document influence only based on texts is a challenging task. Garfield et al. (2002) considered that the impact of a journal is based on aggregate citation counts. To identify influential articles without citations, Gerrish and Blei (2010) proposed the document influence model (DIM), which is a probabilistic model based on the dynamic topic model (Blei and Lafferty, 2006). In DIM, they considered the word frequency change and a document whose words can help the way the word frequencies change will have a high influence score. Gerow et al. (2018) improved DIM by incorporating features, such as authorship, affiliation, and publication venue and they aimed to explain how influence arises. In practice, this additional information is not often available. In this paper, we measure document influence from a more fine-grained level by considering word semantic shifts. Our work differs from the above studies by considering both word frequency changes and word semantic shifts. Specially, we aim to find words that present significant changes in their meanings and we think these words contribute significantly to document influence. Neural-DINF assigns influence scores to documents based on how many of these important words are included in these documents. There has been a lot of research on detecting semantic changes across time (Kay, 1979; Traugott, 1989; Blank, 1999; Zhang et al., 2016; Liao and Cheng, 2016; Bamler and Mandt, 2017). In general, most approaches learn individual embeddings for different time slices and recognize the changes by comparing these embeddings. These vectors have to be aligned into the same vector space for comparison. To achieve alignment, Kim et al. (2014) trained word vectors for different years and then initialized the word vectors in subsequent years with the word vectors obtained from the previous years. Kulkarni et al. (2015) and Hamilton et al. (2016) addressed the embedding alignment problem by learning a linear transformation of words between any two time periods. Most of the alignment methods require anchor words whose meaning does not change between the two time slices. However, it is difficult for us to acquire this kind of prior knowledge, which involves additional expert supervision. In this paper, inspired by Conneau et al. (2017), we propose an adversarial network for unsupervised cross-time alignment. Different from existing approaches, our method is unsupervised and does not require expert information. Our Neural-DINF contains the following three steps. First, we generate static word embeddings in each time slice separately. Then, we implement an unsupervised approach with adversarial training and a refinement procedure to align these embeddings to the same vector space. Finally, we present a new metric to evaluate the influence of a document without citations. Our method first learns individual word embeddings for different time periods and any reasonable word embedding generation approach can be used for this purpose. We consider a text corpus collected across time and use the texts of the documents to train word embeddings. We define our text corpus as D = ( D 1 , . . . , DT ) , where each D t ( t = 1 , . . . , T ) is the texts of all documents in the t -th time slice. The length of these time slices is years in our model. Given any time slice of the texts, our goal is to learn word embeddings through Word2Vec (Mikolov et al., 2013b,a). As our word embeddings for different time periods are trained in different vector spaces, we need to align them to the unified vector space for comparison. We aim at learning a mapping between word vectors for two different time periods. Let S (cid:48) = { s (cid:48) 1 , s (cid:48) 2 , . . . , s (cid:48) m }  R d and S = { s 1 , s 2 , . . . , s n }  R d be two sets of m and n word embeddings from time slices t (cid:48) and t respectively where t (cid:48)  { t +1 , . . . , T } . Ideally, we can use a known dictionary including words that do not experience semantic shifts. Then we can learn a linear mapping W between the two embedding spaces such that: W  = arg min W  R d  d (cid:107) W X  Y (cid:107) 2 , (1) where d is the dimension of the embeddings, and X and Y are two aligned matrices of size d  k formed by k word embeddings selected from S (cid:48) and S , respectively. During the inference time, the aligned embedding of any word w at time slices t (cid:48) is defined as arg max s j T cos( W s (cid:48) w , s j ) . In this paper, we aim to learn this mapping W without using anchor words, which does not change meaning between the two time slices. We first apply an adversarial network to learn an initial proxy of W , then refine the model by using a synthetic parallel dictionary. Domain-Adversarial Training. We define a discriminator which aims at discriminating between elements randomly samples from W S (cid:48) = W s (cid:48) 1 , W s (cid:48) 2 , . . . , W s (cid:48) m and S . The mapping W can be regarded as a generator, which aims at preventing the discriminator from making accurate predictions. The discriminator is designed to maximize its ability to identify the origin of an embedding, and the generator makes W S (cid:48) and S as similar as possible to prevent the discriminator from accurately predicting the embedding origins. We denote the discriminator parameters as  D . Given the mapping W , the optimization objective of the discriminator can be defined as: LD (  D | W ) =  1 m m (cid:88) i =1 log P  D (origin = 1 | W s (cid:48) i )  1 n n (cid:88) j =1 log P  D (origin = 0 | s j ) , (2) where P  D (origin = 1 | z ) is the probability that z originates from the embedding space at time slice t (cid:48) (as opposed to an embedding from the embedding space at time slice t ). The mapping W is trained to prevent the discriminator from accurately predicting embedding origins and the optimization objective can be defined as: LW ( W |  D ) =  1 m m (cid:88) i =1 log P  D (origin = 0 | W s (cid:48) i )  1 n n (cid:88) j =1 log P  D (origin = 1 | s j ) . (3) According to the standard training process of adversarial networks (Goodfellow et al., 2014), the discriminator  D and the mapping W are consecutively trained to respectively minimize LD and LW . Refinement Procedure. The refinement procedure is designed to improve the performance of alignment after the domain-adversarial training step. We obtain a linear transformation W that maps a word from time slices t (cid:48) to t in the last step. To refine our mapping W, we utilize the learned W to build a syntactic parallel dictionary that speci-fies which s (cid:48) i  S (cid:48) refer to which s j  S . Since the most frequent words are suspected to have better embeddings, we consider the most frequent words and keep only their mutual nearest neighbors. In the process of deciding mutual nearest neighbors, we use the Cross-Domain Similarity Local Scaling proposed in (Conneau et al., 2017) to alleviate the hubness problem (Dinu et al., 2014). Consequently, we use Eq. (1) on this obtained dictionary to refine W . To compare vectors from different time periods, we propose an unsupervised approach. An adversarial network is first used to learn an initial proxy of W . To optimize the mapping W , we use a synthetic parallel dictionary in which words' semantics match the best. In this section, Neural-DINF evaluates document influence without citations. Our model makes use of both word frequency changes and word semantic shifts to compute an influence score for each document. We quantify the semantic change of the words by calculating the cosine similarity of the embedding vectors for the same words in different years. We represent aligned vectors of the word w in t and t (cid:48) as w and w (cid:48) respectively. We compute the word meaning shift of w as follows: V w = 1  cos (cid:104) w , w (cid:48) (cid:105) . where D t,t (cid:48) is the vocabulary consisting of co-occurence words of corpus D t and D t (cid:48) , D is the vocabulary of document d , C td,w represents the frequency of word w in the document d , C tw represents the frequency of word w in the corpus D t . The document published at time slice t can only affect documents published after that time slice, so the influence score of document d on the corpus D can be defined as: I d = t (cid:48) = T (cid:88) t (cid:48) = t +1 I t (cid:48) d . Similar to previous studies (Gerrish and Blei, 2010; Gerow et al., 2018) on measuring documents' scholarly impact, we evaluate the performance of Neural-DINF by Pearson correlation and Spearman rank correlation of influence scores and citation counts. We reproduce the DIM (Gerrish and Blei, 2010) as our baseline and its experimental setup is as follows: topics' Markov chain variance  2 = 0 . 005 , topic number K = 5 , LDA (Blei et al., 2003) hyperparameter  = 0 . 001 . In Neural-DINF, word embeddings are generated by training on the corpus of each year and word embedding size is 300. We only select the first 10k most frequent words in each year in our experiments. This threshold is determined by the size of the smallest vocabulary in the years (2002-2013). In the unsupervised alignment, we use the default setting specified in (Conneau et al., 2017) to build a discriminator and the dimension of W is 300  300 . Stochastic gradient descent(SGD) is used to train the discriminator and W with the learning rate of 0.1. We only feed the discriminator with 3000 most frequent words. This is because the embeddings of rare words are of low quality (Luong et al., 2013), which makes them harder to align. It is observed that feeding the discriminator with rare words had a small negative impact which cannot be ignored. In the refinement procedure, we retain the same setting presented in (Conneau et al., 2017). For evaluation, we analyze a sequential corpus The Association for Computational Linguistics Anthology (ACL Anthology), which is a collection of documents on the study of computational linguistics and natural language processing (Bird et al., 2008). Following the experimental setup in DIM, we only use the texts and dates of this corpus. We analyze a subsample from ACL Anthology, spanning from 2002 to 2013, which contains 11106 articles and 18960 unique tokens after preprocessing. We remove short documents and words that have low frequency and low TF-IDF value. Citation counts of articles are obtained from ACL Anthology Network (Joseph and Radev, 2007; Leskovec et al., 2009; Radev et al., 2013). We compare the correlation coefficient scores on DIM and Neural-DINF in Table 1. The Pearson correlation computed by Neural-DINF and DIM is 0.186 and 0.118 respectively. The Spearman rank correlation computed by Neural-DINF and DIM is 0.249 and 0.102 respectively. The results show that our model outperforms the DIM. We also visualize the performances of DIM and our Neural-DINF to validate the effectiveness of our proposed model. As shown in Figure 1, for ACL documents with the highest 60% of influence scores. Neural-DINF covers 83% of citations, which outperforms DIM ( 68% ) by a large marge. In fact, the qualitative analysis does present some evidence that in many cases the Neural-DINF is a better model to produce reasonable scores for the most-cited papers in the used datasets. For example, A Systematic Comparison of Various Statistical Alignment Models (Och and Ney, 2003) is a top-cited article (citation ranking 3) in the dataset. This article receives a very high score both on the DIM and the Neural-DINF. However, the result of Neural-DINF ranking (31) is more close to its citation ranking than the DIM (236). Moreover, in some cases, only Neural-DINF can produce the correct score. For example, DIM assigns a relatively low influence score to (Collins, 2002) (cita-tion ranking 9) in our dataset and ranks this article 11,106 out of 11,106 articles, while the Neural-DINF gives a relatively reasonable score to this article, ranking it 1,199 out of 11,106 articles. In this paper, we aim to evaluate document influence from a fine-grained level by additionally considering word semantic shifts. For our purpose, we develop Neural-DINF which measures document influence from the texts of documents. Besides, we propose an unsupervised method to address the alignment problem. The document receives an influence score based on how it explains the word frequency change and the word semantic shift. Our experimental results show that our model performs better than the DIM on ACL Anthology. This work has been supported in part by National Key Research and Development Program of China (2018AAA010010), NSFC (No.61751209, U1611461), University-Tongdun Technology Joint Laboratory of Artificial Intelligence, Zhejiang University iFLYTEK Joint Research Center, Chinese Knowledge Center of Engineering Science and Technology (CKCEST), China Engineering Expert Tank, Engineering Research Center of Digital Library, Ministry of Education, the Fundamental Research Funds for the Central Universities.
abstain abstain objective abstain method objective method result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain objective objective objective method abstain other other other other other other method objective objective other other other other other other other abstain abstain abstain method method method method method method method method method method method other method method abstain method method abstain method abstain abstain method other other abstain abstain abstain method method method method method abstain method abstain method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain result other