Serial recall experiments study the ability of humans to recall words in the order in which they occurred. The following serial recall effects are generally investigated in studies with humans: word length and frequency , primacy and recency , semantic confusion , repetition , and transposition effects. In this research, we investigate LSTM language models in the context of these serial recall effects. Our work provides a framework to better understand and analyze neural language models and opens a new window to develop accurate language models. The goal of language modeling is to estimate the probability of a sequence of words in natural language, typically allowing one to make probabilistic predictions of the next word given preceding ones (Bahl et al., 1983; Berger et al., 1996). For several years now, Long Short-Term Memory (LSTM) (Graves, 2013) language models have demonstrated state-of-the-art performance (Melis et al., 2018; Merity et al., 2018a; Sundermeyer et al., 2012). Recent studies have begun to shed light on the information encoded by LSTM networks. These models can effectively use distant history (about 200 tokens of context) and are sensitive to word order, replacement, or removal (Khan-delwal et al., 2018), can learn function words much better than content words (Ford et al., 2018), can remember sentence lengths, word identity, and word order (Adi et al., 2017), can capture syntactic structures (Kuncoro et al., 2018) such as subject-verb agreement (Linzen et al., 2016). These characteristics are often attributed to LSTM's ability in overcoming the curse of dimensionality by associating a distributed feature vector to each word (Hinton et al., 1986; Neubig, 2017)and modeling long-range dependencies in faraway context (Khandel-wal et al., 2018). The goal of our research is to complement the prior work to provide a richer understanding about how LSTM language models use prior linguistic context. Inspired by investigations in cognitive psychology about serial recall in humans (Avons et al., 1994; Henson, 1998; Polisenska et al., 2015) where participants are asked to recall a sequence of items in order in which they were presented, we investigate how word length or frequency ( word-frequency effect), word position ( primacy , recency , and transposition effects), word similarity ( semantic confusion effect), and word repetition ( repetition effect) influence learning in LSTM language models. Our investigation provides a framework to better understand and analyze language models at a considerably finer-grained level than previous studies, and opens a new window to develop more accurate language models. We find that LSTM language models (a) can learn frequent/shorter words considerably better than infrequent/longer ones, (b) can learn recent words in sequences better than words in earlier positions, 1 (c) have a tendency to predict words that are semantically similar to target words indicating that these networks have a tendency to group semantically similar words while suggesting one specific word as target based on prior context, (d) predict as output the words that are observed in prior context, i.e. repeat words from prior context, and (e) may transpose (switch adjacent) words in output depending on word syntactic function. 1 Rats and humans recall the first and last items of sequences best and the middle ones worst (Bolhuis and Van Kampen, 1988; Ebbinghaus, 1913). Language models estimate the probability of a sequence as P ( w n 1 ) = (cid:81) ni =1 P ( w i | w i  1 1 ) , where w i is the i th word and w ji indicates the sub-sequence from w i to w j . These models minimize their prediction error against words in context, using e.g. the negative log likelihood loss function, during training: L =  1 n  ni =1 log P ( w i | w i  1 1 ) . In this paper, we show the loss of a model against sequence s and word w i  s by L ( s ) and L ( s )[ w i ] respectively. We use the LSTM language model developed in (Merity et al., 2018b) for our experiments. Given a sequence of words w i  1 1 as context, this model predicts the next word in sequence. We refer to w i and  w i as target and predicted words respectively; given the global vocabulary V ,  w i = arg max w j V Pr( w j | w i  1 1 ) . We study this LSTM in the context of serial recall effects. What is the effect of word frequency/length 2 on the performance of LSTM language models? For this effect, we report the average loss for each word frequency as follows: L kWF = 1 / |S k | (cid:88) s S k ,w i  s L ( s )[ w i ] , (1) where S k is the set of sequences that, at least, have one target word with term frequency of k , and L kWF is the overall loss for target words of frequency k which sheds light on the expected frequency of words for accurate language modeling. What is the effect of word position on the performance of LSTM language models? To analyze this effect, we compute the average loss of network with respect to the position of target words as follows: L iPR = 1 /Z (cid:88) s L ( s )[ w i ] , (2) where w i is the target word in sequence s , Z is the number of sequences as normalization factor, and L iPR is the average of loss at position i . This effect will shed light on network performance at specific positions in texts which can help rationalizing the need for new language modeling architectures, such as bidirectional LSTMs (Schuster and Paliwal, 1997), or the order by which input data should be processed (Sutskever et al., 2014). Are predicted words semantically similar to the target ones in case of incorrect predictions? For this analysis, we report the average semantic similarity between target ( w i ) and predicted (  w i ) words as follows: SC = 1 /Z (cid:88) s,w i  s,w i (cid:54) =  w i sim ( w i ,  w i ) , (3) where the function sim ( ., . ) computes word similarity either through WordNet (Miller, 1998) or cosine similarity of the corresponding embeddings of its arguments. This effect will shed light on how effective LSTMs are in disentangling semantically-similar concepts. This gives us a powerful metric to compare networks semantically, especially, in case of equal loss/perplexity. This effect refers to prediction of a word that already exists in context, i.e. in an earlier position in the sequence. Here, for each target w n , we compute the probability that, instead of w n , any of w n  1 1 words is predicted as output and report average across all samples: Pr( RE i ) = (4) 1 /Z (cid:88) s Pr( w i  w n  1 1 , w i (cid:54) = w n | w n  1 1 ) . This effect will shed light on the extent to which network repeats/predicts observed words as possible responses. Given that words rarely repeat in sentences, the above metric can be used as a good regularizer for language modeling. This effect refers to word prediction in transposed positions, i.e. the case where the word pair w i +1 i in an original sequence is more likely to be predicted by the network as w i +1 w i in output. Here, for each pair w i +1 i in target sequence, we count the number of times in which w i +1 is more probable to be predicted at position i (as compared to w i ) and w i is more probable to be predicted at position i +1 (as compared to w i +1 ). We report the average number of transposition occurrences for all samples at each word position. This effect will shed light on how network learn nearby grammatical orders such as conjunction and adjective order. 258 259 260 Figures 1a and Figures 1b show this effect in POS level for both PTB and WT2. It shows the strong correlation between word length and loss in content words while function words are not so affected. (a) Dataset statistics, (b): Word-frequency effect, network learns frequent words better than infrequent ones, (c): Primacy and Recency effect, loss decreases for target words that appear at the end of sequences. 264 265 266 267 268 269 270 271 272 273 274 Figure 1c reveals the strong negative correlation between word frequency and loss such that increase in word frequency leads to decrease in loss. From another viewpoint, Figure 1d shows that the frequency of content words decrease while their length increase but Figure 1e shows frequency of function words is independent of their length. The fact that function words are closed class and we can not extend them with shorter words can be the reason. It is also reported in (Bell et al., 2009) that content words are shorter when more frequent while function words are not so. Therefore, word length effect is strongly exists in content words but is weak in function words. 3 Experiments Datasets . We use two benchmark language modeling datasets: Penn Treebank (PTB) (Marcus et al., 1993; Mikolov et al., 2010) and WikiText-2 (WT2) (Merity et al., 2017). PTB and WT2 have vocabulary sizes of 10 K and 33 K respectively. We use the POS-tagged versions of these datasets provided by Khandelwal et al. (2018), and treat nouns (NN), verbs (VB), adjectives (JJ), and adverbs (RB) as content words, and others word classes as function words, see details in Figure 1a. 275 276 277 278 279 280 281 282 283 Also, from Figure 1c and the fact that average frequency in function words is much higher than content words ( 10k in compare to 58 in PTB dataset), it is now clear that why loss of function words are significantly lower than content words. Tokens later in the sequence have better recall. We examine primacy and recency effect by comparing the loss values in different positions of the sequence. Settings . We set LSTM's parameters as suggested in (Merity et al., 2018b) for PTB and follow its suggested parameter tuning procedure for WT2. For both datasets, we set context size to n = 100 obtained from { 5 , 20 , 50 , 100 , 200 } and validation data; note that the number of samples are equal for different sequence lengths. 285 286 287 Figure 2 shows the loss decreases (i.e. the network has better recall) as we approach the end of the sequence. 3.1 Results We report LSTM performance in terms of prediction loss on development sets for all experiments. 288 289 290 291 292 293 294 295 296 297 298 299 Recalling words in transposed positions are fairly rare. To figure out the transposition effect, we count how many times words recall in transposed positions. Table 1 shows the number of transpositions normalized over sequence length. It reveals that number of transpositions do not increase while the sequence length increases. Moreover, transposition occurs rarely. For example, in sequence of length 100, from 99 candidates of transposition only 0.72 transposed on average. Figure 3 shows the number of transposition occurrences in each position of this sequence. It shows that Word-Frequency Effect : More frequent target words are predicted (learned) more accurately than less frequent ones. Figure 1b shows strong inverse correlation between word frequency and LSTM prediction loss. This is expected as neural models learn better with more data. In addition, although the overall loss of function words is considerably lower than that of content words (because of their overall higher frequency), Figure 1b shows that, for the same word frequency, content words are learned better than function words. Primacy and Recency Effects : Target words that appear later in sequences are predicted considerably better than those at earlier positions. Figure 1c shows that prediction loss considerably decreases for target words that appear toward the end of the sequences. The results are consistent across 1 3 5 7 9 11 Loss ( ) 0.2 0.4 0.6 0.8 1.0 s i m ( t a r g e t , p r e d i c t e d ) Embedding w nearest neithbor random word 1 3 5 7 9 11 Loss ( ) WordNet both datasets. This effect can explain why bidirectional LSTMs which read input from opposite directions usually work better in NLP applications such as machine translation (Firat et al., 2016; Domhan, 2018). A remaining question that is worth investigating is whether bidirectional LSTMs learn first and last few words of sequences better than those in the middle, and if yes, how can we make these models more robust against word position. Semantic Confusion Effect . There is signifi-cant tendency to predict words that resemble (are semantically-similar to) target words. Figure 2 shows the average WordNet and Embedding similarity between target and predicted words in PTB and WT2 across loss values. The results indicate high similarity between predicted and target words target predicted 0.0 0.2 0.4 0.6 0.8 1.0 p r o b a b ili t y predicted similar neighbours others predicted similar neighbours others for smaller loss values. However, confusion consistently increases as prediction loss increases. The upper bound similarity (obtained by treating the nearest neighbor of each target word as predicted word) indicates there exists better candidates which LSTM fails to predict. Our further analyses show that LSTM has a tendency to group semantically similar words and then suggest one of them. We consider most similar words as a group of neighbors and examine how network assigns probabilities to them as compared to others. Figure 3a shows that the chance of neighbors of target (with size 150) is equal to chance of other words (with size 9849). As Figure 3a shows, the neighbors of predicted words (with size 600) carry equal chance as compared to other words (with size 9399). To find these thresholds, we gradually increase the number of neighbors and track the trend of approaching the probability of neighboring group to target. As shown in Figure 3b, if the size of neighboring group is set to 150, these probabilities became equal. The similar way is repeated for finding the appropriate size for neighbors of predicted words. content words . We report repetition effect, see Eq. (4), at POS tag level (where the predicted word should have the same POS tag as target word in prior context). As Figure 4a and 4b show, function words have higher repetition probability than content words. This is because function words are more frequent, and the average distance among them (i.e. number of intervening words) is considerably smaller than other POS tags (e.g. 2 . 1 words vs 28 . 9 and 12 . 4 words for RB and JJ respectively). We also find that repetition probability decreases as a function of word frequency in prior context, see Figure 4c. This is because words (especially NN s and VB s) are often self-contained and their occurrence in prior context helps LSTM to avoid repeating them. In addition, we find that function words repeat more frequently than other types and repetition among NN s and VB s is higher than other POS tag pairs; Table 1 shows the confusion matrix for repetition across POS tag classes Perhaps, this could explain the recent language modeling improvement obtained in (Ford et al., 2018) through developing separate yet sequentially connected network architectures with respect to POS tag class. There are two factors determining the chance of 5 10 20 50 100 200 context size (number of tokens) 0.5 1.0 1.5 2.0 L o ss POSJJ NN RB VB func PTB WT2 Figure 5: Effective context to learn syntax. repetition of a POS class: First, the average distance of consecutive tokens of that POS class; Table 2 reports the corresponding values from training set. Second, the accuracy of network in predicting POS classes which has been shown in Figure 5 and also reported in (Khandelwal et al., 2018). From these, the repetition probability of function words are expected to be higher than content words. Transposition Effect : Transpositions occur more frequently at the beginning of sequences and rarely at the end . Figure 6 shows average number of transpositions at each word position across datasets. This result is meaningful because miss-predictions occur more frequently at earlier positions (see results for primacy and recency effect). In addition, transpositions are rare at higher positions because more context information can help LSTM to make accurate (transposition-free) prediction. In addition, Table 3 shows the percentage of transpositions across POS tag classes on PTB. The result show that LSTM mainly transposes  RB NN ' word pairs with  NN RB .' In future, we will conduct further analyses to understand the reason. The findings presented in this paper provide insight into how LSTMs model context. This information can be useful for improving language mod-0 20 40 60 80 100 position in sequence 500 550 600 650 700 750 1.2k 1.5k 1.7k 2k 2.2k 2.5k # t r a n s p o s i t i o n s PTB # t r a n s p o s i t i o n s WT 2 PTB WT2 Figure 6: Transposition effect, transpositions occur more at the beginning of sequences. els. For instance, the discovery that some word types are repeated in predictions more than others can help regularizing neural language models by making them adaptive to the different word types. We investigate LSTM language models in the context of serial recall indicators. We find that frequent target words and target words that appear later in sequences are predicted more accurately, predictions often resemble (are semantically similar to) target words, function words of prior context are more likely to be predicted as target words, and word pair transpositions occur more frequently at the beginning of sequences. We sincerely thank Nazanin Dehghani for her constructive collaboration during development of this paper and anonymous reviewers for their insightful comments and effective feedback.
abstain abstain objective objective abstain abstain abstain other abstain objective other objective result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method method abstain method abstain method abstain abstain method abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective result other