How does the input segmentation of pretrained language models (PLMs) affect their interpretations of complex words? We present the first study investigating this question, taking BERT as the example PLM and focusing on its semantic representations of English derivatives. We show that PLMs can be interpreted as serial dual-route models, i.e., the meanings of complex words are either stored or else need to be computed from the subwords, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis is confirmed by a series of semantic probing tasks on which DelBERT (Derivation leveraging BERT), a model with derivational input segmentation, substantially outperforms BERT with WordPiece segmentation. Our results suggest that the generalization capabilities of PLMs could be further improved if a morphologically-informed vocabulary of input tokens were used. Pretrained language models (PLMs) such as BERT (Devlin et al., 2019), GPT-2 (Radford et al., 2019), XLNet (Yang et al., 2019), ELECTRA (Clark et al., 2020), and T5 (Raffel et al., 2020) have yielded substantial improvements on a range of NLP tasks. What linguistic properties do they have? Various studies have tried to illuminate this question, with a focus on syntax (Hewitt and Manning, 2019; Jawa-har et al., 2019) and semantics (Ethayarajh, 2019; Ettinger, 2020; Vulic et al., 2020). One common characteristic of PLMs is their input segmentation: PLMs are based on fixed-size vocabularies of words and subwords that are generated by compression algorithms such as byte-pair encoding (Gage, 1994; Sennrich et al., 2016) and WordPiece (Schuster and Nakajima, 2012; Wu et al., 2016). The segmentations produced by these s w x y superbizarre neg applausive pos ##iza superb ##rre BERT p ( y | s w ( x )) = . algorithms are linguistically questionable at times (Church, 2020), which has been shown to worsen performance on certain downstream tasks (Bostrom and Durrett, 2020; Hofmann et al., 2020a). However, the wider implications of these findings, particularly with regard to the generalization capabilities of PLMs, are still poorly understood. Here, we address a central aspect of this issue, namely how the input segmentation affects the semantic representations of PLMs, taking BERT as the example PLM. We focus on derivationally complex words such as superbizarre since they exhibit systematic patterns on the lexical level, providing an ideal testbed for linguistic generalization. At the same time, the fact that low-frequency and out-of-vocabulary words are often derivationally complex (Baayen and Lieber, 1991) makes our work relevant in practical settings, especially when many one-word expressions are involved, e.g., in query processing (Kacprzak et al., 2017). The topic of this paper is related to the more fundamental question of how PLMs represent the meaning of complex words in the first place. So far, most studies have focused on methods of representation extraction, using ad-hoc heuristics such as averaging the subword embeddings (Pinter et al., 2020; Sia et al., 2020; Vulic et al., 2020) or taking the first subword embedding (Devlin et al., 2019; Heinzerling and Strube, 2019; Martin et al., 2020). While not resolving the issue, we lay the theoretical groundwork for more systematic analyses by showing that PLMs can be regarded as serial dual-route models (Caramazza et al., 1988), i.e., the meanings of complex words are either stored or else need to be computed from the subwords. Contributions. We present the first study examining how the input segmentation of PLMs, specifically BERT, affects their interpretations of derivationally complex English words. We show that PLMs can be interpreted as serial dual-route models, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis is confirmed by a series of semantic probing tasks on which derivational segmentation substantially outperforms BERT's WordPiece segmentation. This suggests that the generalization capabilities of PLMs could be further improved if a morphologically-informed vocabulary of input tokens were used. We also publish three large datasets of derivationally complex words with corresponding semantic properties. 1 2 How Are Complex Words Processed? The question of how complex words are processed has been at the center of psycholinguistic research over the last decades (see Leminen et al. (2019) for a recent review). Two basic processing mechanisms have been proposed: storage , where the meaning of complex words is listed in the mental lexicon (Manelis and Tharp, 1977; Butterworth, 1983; Feldman and Fowler, 1987; Bybee, 1988; Stemberger, 1994; Bybee, 1995; Bertram et al., 2000a), and computation , where the meaning of complex words is inferred based on the meaning of stem and affixes (Taft and Forster, 1975; Taft, 1979, 1981, 1988, 1991, 1994; Rastle et al., 2004; Taft, 2004; Rastle and Davis, 2008). In contrasting with single-route frameworks, dual-route models allow for a combination of storage and computation. Dual-route models are further classified by whether they regard the processes of retrieving meaning from the mental lexicon and computing meaning based on stem and affixes as parallel , i.e., both mechanisms are always activated (Frauenfelder and Schreuder, 1992; Schreuder and Baayen, 1995; Baayen et al., 1997, 2000; Bertram et al., 2000b; New et al., 2004; Kuperman et al., 2008, 2009), or serial , i.e., the computation-based mechanism is only activated when the storage-based one fails (Laudanna and Burani, 1985; Burani and Caramazza, 1987; Caramazza et al., 1988; Burani and Laudanna, 1992; Laudanna and Burani, 1995; Alegre and Gordon, 1999). Outside the taxonomy presented so far are recent models that assume multiple levels of representation as well as various forms of interaction between them (Rcz et al., 2015; Needle and Pierrehumbert, 2018). In these models, sufficiently frequent complex words are stored together with representations that include their internal structure. Complex-word processing is driven by analogical processes over the mental lexicon (Rcz et al., 2020). Most models of word meaning proposed in NLP can be roughly assigned to either the single-route or dual-route approach. Word embeddings that represent complex words as whole-word vectors (Deerwester et al., 1990; Mikolov et al., 2013a,b; Pennington et al., 2014) can be seen as single-route storage models. Word embeddings that represent complex words as a function of subword or morpheme vectors (Schtze, 1992; Luong et al., 2013) can be seen as single-route computation models. Finally, word embeddings that represent complex words as a function of subword or morpheme vectors as well as whole-word vectors (Botha and Blun-som, 2014; Qiu et al., 2014; Bhatia et al., 2016; Bojanowski et al., 2017; Athiwaratkun et al., 2018; Salle and Villavicencio, 2018) are most closely related to parallel dual-route approaches. Where are PLMs to be located in this taxonomy? PLMs represent many complex words as whole-word vectors (which are fully stored). Similarly to how character-based models represent word meaning (Kim et al., 2016; Adel et al., 2017), they can also store the meaning of frequent complex words that are segmented into subwords, i.e., frequent subword collocations, in their model weights. When the complex-word meaning is neither stored as a whole-word vector nor in the model weights, PLMs compute the meaning as a compositional function of the subwords. Conceptually, PLMs can thus be interpreted as serial dual-route models. While the parallelism has not been observed before, it follows logically from the structure of PLMs. The key goal of this paper is to show that the implications of this observation are borne out empirically. As a concrete example, consider the complex words stabilize , realize , finalize , mobilize , tribalize , and templatize , which are all formed by adding the verbal suffix ize to a nominal or adjectival stem. Taking BERT, specifically BERTBASE (uncased) (De-vlin et al., 2019), as the example PLM, the words stabilize and realize have individual tokens in the input vocabulary and are hence associated with whole-word vectors storing their meanings, including highly lexicalized meanings as in the case of realize . By contrast, the words finalize and mobilize are segmented into final , ##ize and mob , ##ili , ##ze , which entails that their meanings are not stored as whole-word vectors. However, both words have relatively high absolute frequencies of 2,540 ( finalize ) and 6,904 ( mobilize ) in the English Wikipedia, the main dataset used to pretrain BERT (Devlin et al., 2019), which means that BERT can store their meanings in its model weights during pretraining. 2 Notice this is even possible in the case of highly lexicalized meanings as for mobilize . Finally, the words tribalize and templatize are segmented into tribal , ##ize and te , ##mp , ##lat , ##ize , but as opposed to finalize and mobilize they do not occur in the English Wikipedia. As a result, BERT cannot store their meanings in its model weights during pretraining and needs to compute them from the meanings of the subwords. Seeing PLMs as serial dual-route models allows for a more nuanced view on the central research question of this paper: in order to investigate semantic generalization we need to investigate the representations of those complex words that activate the computation-based route. The words that do so are the ones whose meaning is neither stored as a whole-word vector nor in the model weights 2 Previous research suggests that such lexical knowledge is stored in the lower layers of BERT (Vulic et al., 2020). and hence needs to be computed compositionally as a function of the subwords ( tribalize and templatize in the discussed examples). We hypothesize that the morphological validity of the segmentation affects the representational quality in these cases, and that the best generalization is achieved by maximally meaningful tokens. It is crucial to note this does not imply that the tokens have to be morphemes, but the segmentation boundaries need to coincide with morphological boundaries, i.e., groups of morphemes (e.g., tribal in the segmentation of tribalize ) are also possible. 3 For tribalize and templatize , we therefore expect the segmentation tribal , ##ize (morphologically valid since all segmentation boundaries are morpheme boundaries) to result in a representation of higher quality than the segmentation te , ##mp , ##lat , ##ize (morpho-logically invalid since the boundaries between te , ##mp , and ##lat are not morpheme boundaries). On the other hand, complex words whose meanings are stored in the model weights ( finalize and mobilize in the discussed examples) are expected to be affected by the segmentation to a much lesser extent: if the meaning of a complex word is stored in the model weights, it should matter less whether the specific segmentation activating that meaning is morphologically valid ( final , ##ize ) or not ( mob , ##ili , ##ze ). 4 3 Experiments 3.1 Setup Analyzing the impact of different segmentations on BERT's semantic generalization capabilities is not straightforward since it is not clear a priori how to measure the quality of representations. Here, we devise a novel lexical-semantic probing task: we use BERT's representations for complex words to predict semantic dimensions, specifically sentiment and topicality (see Figure 1). For sentiment, given the example complex word superbizarre , the task is to predict that its sentiment is negative. For topicality, given the example complex word isotopize , the task is to predict that it is used in physics. We confine ourselves to binary predic-3 This is in line with substantial evidence from linguistics showing that frequent groups of morphemes can be treated as semantic wholes (Stump, 2017, 2019). 4 We expect the distinction between storage and computation of complex-word meaning for PLMs to be a continuum. While the findings presented here are consistent with this view, we defer a more in-depth analysis to future work. tion, i.e., the probed semantic dimensions always consist of two classes (e.g., positive and negative). The extent to which a segmentation supports a solution of this task is taken as an indicator of its representational quality. More formally, let D be a dataset consisting of complex words x and corresponding classes y that instantiate a certain semantic dimension (e.g., sen-timent). We denote with s ( x ) = ( t 1 , . . . , t k ) the segmentation of x into a sequence of k subwords. We ask how s impacts the capability of BERT to predict y , i.e., how p ( y | ( s ( x )) , the likelihood of the true semantic class y given a certain segmentation of x , depends on different choices for s . The two segmentation methods we compare in this study are BERT's standard WordPiece segmentation (Schus-ter and Nakajima, 2012; Wu et al., 2016), s w , and a derivational segmentation that segments complex words into stems and affixes, s d . Since existing datasets do not allow us to conduct experiments following the described setup, we create new datasets in a weakly-supervised fashion that is conceptually similar to the method proposed by Mintz et al. (2009): we employ large datasets annotated for sentiment or topicality, extract derivationally complex words, and use the dataset labels to establish their semantic classes. For determining and segmenting derivationally complex words, we use the algorithm introduced by Hofmann et al. (2020b), which takes as input a set of prefixes, suffixes, and stems and checks for each word in the data whether it can be derived from a stem using a combination of prefixes and suffixes. 5 The algorithm is sensitive to morpho-orthographic rules of English (Plag, 2003), e.g., when the suf-5 The distinction between inflectionally and derivationally complex words is notoriously fuzzy (Haspelmath and Sims, 2010; ten Hacken, 2014). We try to exclude inflection as far as possible (e.g., by removing problematic affixes such as ing ) but are aware that a clear separation does not exist. fix ize is removed from isotopize , the result is isotope , not isotop . We follow Hofmann et al. (2020a) in using the prefixes, suffixes, and stems in BERT's WordPiece vocabulary as input to the algorithm. This means that all tokens used by the derivational segmentation are in principle also available to the WordPiece segmentation, i.e., the difference between s w and s d does not lie in the vocabulary per se but rather in the way the vocabulary is used. See Appendix A.1 for details about the derivational segmentation. To get the semantic classes, we compute for each complex word which fraction of texts containing the word belongs to one of two predefined sets of dataset labels (e.g., reviews with four and five stars for positive sentiment) and rank all words accordingly. We then take the first and third tertiles of complex words as representing the two classes. We randomly split the words into 60% training, 20% development, and 20% test. In the following, we describe the characteristics of the three datasets in greater depth. Table 1 provides summary statistics. See Appendix A.2 for details about data preprocessing. Amazon. Amazon is an online e-commerce platform. A large dataset of Amazon reviews has been made publicly available (Ni et al., 2019). 6 We extract derivationally complex words from reviews with one or two ( neg ) as well as four or five stars ( pos ), discarding three-star reviews for a clearer separation (Yang and Eisenstein, 2017). ArXiv. ArXiv is an open-access distribution ser-vice for scientific articles. Recently, a dataset of all papers published on ArXiv with associated meta-data has been released. 7 For this study, we extract all articles from physics ( phys ) and computer science ( cs ), which we identify using ArXiv's subject classification. We choose physics and computer 6 https://nijianmo.github.io/amazon/ index.html 7 https://www.kaggle.com/ Cornell-University/arxiv Amazon ArXiv Reddit Model Dev Test Dev Test Dev Test DelBERT .635  .001 .639  .002 .731  .001 .723  .001 .696  .001 .701  .001 BERT .619  .001 .624  .001 .704  .001 .700  .002 .664  .001 .664  .003 Stem .572  .003 .573  .003 .705  .001 .697  .001 .679  .001 .684  .002 Affixes .536  .008 .539  .008 .605  .001 .603  .002 .596  .001 .596  .001 Table 2: Results. science since we expect large topical distances for these classes (compared to alternatives such as mathematics and computer science). Reddit. Reddit is a social media platform hosting discussions about various topics. It is divided into smaller communities, so-called subreddits, which have been shown to be a rich source of derivationally complex words (Hofmann et al., 2020c). Hofmann et al. (2020a) have published a dataset of derivatives found on Reddit annotated with the subreddits in which they occur. 8 Inspired by a content-based subreddit categorization scheme, 9 we define two groups of subreddits, an entertainment set ( ent ) consisting of the subreddits anime , DestinyTheGame , funny , Games , gaming , leagueoflegends , movies , Music , pics , and videos , as well as a discussion set ( dis ) consisting of the subred-8 https://github.com/valentinhofmann/ dagobert 9 https://www.reddit.com/r/ TheoryOfReddit/comments/1f7hqc/the_200_most_active_subreddits_categorized_by dits askscience , atheism , conspiracy , news , Libertarian , politics , science , technology , TwoXChromosomes , and worldnews , and extract all derivationally complex words occurring in them. We again expect large topical distances for these classes. Given that the automatic creation of the datasets necessarily introduces noise, we measure human performance on 100 randomly sampled words per dataset, which ranges between 71% (Amazon) and 78% (ArXiv). These values can thus be seen as an upper bound on performance. We train two main models on each binary classi-fication task: BERT with the standard WordPiece segmentation ( s w ) and BERT using the derivational segmentation ( s d ), a model that we refer to as DelBERT ( De rivation l everaging BERT ). BERT and DelBERT are identical except for the way in which they use the vocabulary of input tokens (but the vocabulary itself is also identical for both models). The specific BERT variant we use is BERTBASE (uncased) (Devlin et al., 2019). For the derivational segmentation, we follow previous work by Hofmann et al. (2020a) in separating stem and prefixes by a hyphen. We further follow Casanueva et al. (2020) and Vulic et al. (2020) in mean-pooling the output representations for all subwords, excluding BERT's special tokens. The mean-pooled representation is then fed into a two-layer feed-forward network for classification. To examine the relative importance of different types of morphological units, we train two additional models in which we ablate information about stems and affixes, i.e., we represent stems and affixes by the same randomly chosen input embedding. 10 We finetune BERT, DelBERT, and the two ablated models on the three datasets using 20 different random seeds. We choose F1 as the evaluation measure. See Appendix A.3 for details about implementation and hyperparameters. DelBERT ( s d ) outperforms BERT ( s w ) by a large margin on all three datasets (Table 2). It is interesting to notice that the performance difference is larger for ArXiv and Reddit than for Amazon, indicating that the gains in representational quality are particularly large for topicality. What is it that leads to DelBERT's increased performance? The ablation study shows that models using only stem information already achieve relatively high performance and are on par or even better than the BERT models on ArXiv and Reddit. However, the DelBERT models still perform substantially better than the stem models on all three datasets. The gap is particularly pronounced 10 For affix ablation, we use two different input embeddings for prefixes and suffixes. for Amazon, which indicates that the interaction between the meaning of stem and affixes is more complex for sentiment than for topicality. This makes sense from a linguistic point of view: while stems tend to be good cues for the topical associations of a complex word, sentiment often depends on semantic interactions between stems and affixes. For example, while the prefix un turns the sentiment of amusing negative, it turns the sentiment of biased positive. Such effects involving negation and antonymy are known to be challenging for PLMs (Ettinger, 2020; Kassner and Schtze, 2020) and might be one of the reasons for the generally lower performance on Amazon. 11 The performance of models using only affixes is much lower. To further examine how BERT ( s w ) and DelBERT ( s d ) differ in the way they infer the meaning of complex words, we perform a convergence analysis. We find that the DelBERT models reach their peak in performance faster than the BERT models (Figure 2). This is in line with our interpretation of PLMs as serial dual-route models (see Section 2.2): while DelBERT operates on morphological units and can combine the subword meanings to infer the meanings of complex words, BERT's subwords do not necessarily carry lexical meanings, and hence the derivational patterns need to be stored by adapting the model weights. This is an additional burden, leading to longer convergence times and substantially worse overall performance. 11 Another reason for the lower performance on sentiment is that the datasets were created automatically (see Section 3.2), and hence many complex words do not directly carry information about sentiment or topicality. The density of such words is higher for sentiment than topicality since the topic of discussion affects the likelihoods of most content words. to process complex words (storage in weights and compositional computation based on input embed-dings), and that the second route is blocked when the input segmentation is not morphological, suggests the existence of frequency effects: BERT might have seen frequent complex words multiple times during pretraining and stored their meaning in the model weights. This is less likely for infrequent complex words, making the capability to compositionally infer the meaning (i.e., the computation route) more important. We therefore expect the difference in performance between DelBERT (which should have an advantage on the computation route) and BERT to be larger for infrequent words. To test this hypothesis, we split the complex words of each dataset into three bins of low ( f  5 ), mid ( 5 < f  500 ), and high ( f > 500 ) absolute frequencies, and analyze how the performance of BERT and DelBERT differs on the three bins. For this and all subsequent analyses, we merge development and test sets and use accuracy instead of F1 since it makes comparisons across small sets of data points more interpretable. The results are in line with our hypothesis (Figure 3): BERT performs worse than DelBERT on complex words of low and mid frequencies but achieves very similar (ArXiv, Reddit) or even better (Amazon) accuracies on high-frequency complex words. These results strongly suggest that two different mechanisms are involved, and that BERT has a disadvantage for complex words that do not have a high frequency. At the same time, the slight advantage of BERT on high-frequency complex words indicates that it has high-quality representations of these words in its weights, which DelBERT cannot exploit since it uses a different segmentation. We are further interested to see whether the affix type has an impact on the relative performance of BERT and DelBERT. To examine this question, we measure the accuracy increase of DelBERT as compared to BERT for individual affixes, averaged across datasets and random seeds. We find that the increase is almost twice as large for prefixes (  = . 023 ,  = . 017 ) than for suffixes (  = . 013 ,  = . 016 ), a difference that is shown to be significant by a two-tailed Welch's t -test ( d = . 642 , t (82 . 97) = 2 . 94 , p < . 01 ). 12 Why is having access to the correct morphological segmentation more advantageous for prefixed than suffixed complex words? We argue that there are two key factors at play. First, the WordPiece tokenization sometimes generates the morphologically correct segmenta-12 We use a Welch's instead of Student's t -test since it does not assume that the distributions have equal variance. tion, but it does so with different frequencies for prefixes and suffixes. To detect morphologically incorrect segmentations, we check whether the WordPiece segmentation keeps the stem intact, which is in line with our definition of morphological validity (Section 2.2) and provides a conservative estimate of the error rate. For prefixes, the WordPiece tokenization is seldom correct (average error rate:  = . 903 ,  = . 042 ), whereas for suffixes it is correct about half the time (  = . 503 ,  = . 213 ). Hence, DelBERT gains a greater advantage for prefixed words. Second, prefixes and suffixes have different linguistic properties that affect the prediction task in unequal ways. Specifically, whereas suffixes have both syntactic and semantic functions, prefixes have an exclusively semantic function and always add lexical-semantic meaning to the stem (Giraudo and Grainger, 2003; Beyersmann et al., 2015). As a result, cases such as unamusing where the affix boundary is a decisive factor for the prediction task are more likely to occur with prefixes than suffixes, thus increasing the importance of a morphologically correct segmentation. 13 Given the differences between sentiment and topicality prediction, we expect variations in the relative importance of the two identified factors: (i) in the case of sentiment the advantage of s d should be maximal for affixes directly affecting sentiment; (ii) in the case of topicality its advantage should be the larger the higher the proportion of incorrect segmentations for a particular affix, and hence the more frequent the cases where DelBERT has access to the stem while BERT does not. To test this hypothesis, we focus on pre-13 Notice that there are suffixes with similar semantic effects (e.g., less ), but they are less numerous. dictions for prefixed complex words. For each dataset, we measure for individual prefixes the accuracy increase of the DelBERT models as compared to the BERT models, averaged across random seeds, as well as the proportion of morphologically incorrect segmentations produced by WordPiece. We then calculate linear regressions to predict the accuracy increases based on the proportions of incorrect segmentations. This analysis shows a significant positive correlation for ArXiv ( R 2 = . 304 , F (1 , 41) = 17 . 92 , p < 0 . 001 ) and Reddit ( R 2 = . 270 , F (1 , 40) = 14 . 80 , p < 0 . 001 ) but not for Amazon ( R 2 = . 019 , F (1 , 41) = . 80 , p = . 375 ), which is in line with our expectations (Figure 4a). Furthermore, ranking the prefixes by accuracy increase for Amazon confirms that the most pronounced differences are found for prefixes that can change the sentiment such as non , anti , mal , and pseudo (Figure 4b). Besides quantitative factors, we are interested in identifying qualitative contexts in which DelBERT has a particular advantage compared to BERT. To do so, we filter the datasets for complex words that are consistently classified correctly by DelBERT and incorrectly by BERT. Specifically, we compute for each word the average likelihood of the true semantic class across DelBERT and BERT models, respectively, and rank words according to the likelihood difference between both model types. Examining the words with the most extreme differences, we observe three classes (Table 3). First, the addition of a suffix is often connected with morpho-orthographic changes (e.g., the deletion of a stem-final e ), which leads to a segmentation of the stem into several subwords since the truncated stem is not in the WordPiece vocabulary ( applausive , isotopize , prematuration ). The model does not seem to be able to recover the meaning of the stem from the subwords. Second, the addition of a prefix has the effect that the word-internal (as opposed to word-initial) form of the stem would have to be available for proper segmentation. Since this form rarely exists in the WordPiece vocabulary, the stem is segmented into several subwords ( superannoying , antimicrosoft , nonmultiplayer ). Again, it does not seem to be possible for the model to recover the meaning of the stem. Third, the segmentation of prefixed complex words often fuses the prefix with the first characters of the stem ( overseasoned , inkinetic , promosque ). This case is particularly detrimental since it not only makes it difficult to recover the meaning of the stem but also creates associations with unrelated meanings, sometimes even opposite meanings as in the case of superbizarre . The three classes thus underscore the difficulty of inferring the meaning of complex words from the subwords when the whole-word meaning is not stored in the model weights and the subwords are not morphological. Several recent studies have examined how the performance of PLMs is affected by their input segmentation. Tan et al. (2020) show that tokenizing inflected words into stems and inflection symbols allows BERT to generalize better on non-standard inflections. Bostrom and Durrett (2020) pretrain RoBERTa with different tokenization methods and find tokenizations that align more closely with morphology to perform better on a number of tasks. Ma et al. (2020) show that providing BERT with character-level information also leads to enhanced performance. Relatedly, studies from automatic speech recognition have demonstrated that morphological decomposition improves the perplexity of language models (Fang et al., 2015; Jain et al., 2020). Whereas these studies change the vocabulary of input tokens (e.g., by adding special tokens), we show that even when keeping the pretrained vocabulary fixed, employing it in a morphologically correct way leads to better performance. 14 14 There are also studies that analyze morphological aspects of PLMs without a focus on questions surrounding segmentation (Edmiston, 2020; Klemen et al., 2020). Most NLP studies on derivational morphology have been devoted to the question of how semantic representations of derivationally complex words can be enhanced by including morphological information (Luong et al., 2013; Botha and Blun-som, 2014; Qiu et al., 2014; Bhatia et al., 2016; Cotterell and Schtze, 2018), and how affix embeddings can be computed (Lazaridou et al., 2013; Kisselew et al., 2015; Pad et al., 2016). Cotterell et al. (2017), Vylomova et al. (2017), and Deutsch et al. (2018) propose sequence-to-sequence models for the generation of derivationally complex words. Hofmann et al. (2020a) address the same task using BERT. In contrast, we analyze how different input segmentations affect the semantic representations of derivationally complex words in PLMs, a question that has not been addressed before. We have examined how the input segmentation of PLMs, specifically BERT, affects their interpretations of derivationally complex words. Drawing upon insights from psycholinguistics, we have deduced a conceptual interpretation of PLMs as serial dual-route models, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis was confirmed by a series of semantic probing tasks on which DelBERT, a model using derivational segmentation, consistently outperformed BERT using WordPiece segmentation. Quantitative and qualitative analyses further showed that BERT's inferior performance was caused by its inability to infer the complex-word meaning as a function of the subwords when the complex-word meaning was not stored in the weights. Overall, our findings suggest that the generalization capabilities of PLMs could be further improved if a morphologically-informed vocabulary of input tokens were used. This work was funded by the European Research Council (#740516) and the Engineering and Physical Sciences Research Council (EP/T023333/1). The first author was also supported by the German Academic Scholarship Foundation and the Arts and Humanities Research Council. We thank the reviewers for their helpful comments.
abstain objective objective abstain result abstain abstain abstain abstain abstain abstain abstain method method abstain objective abstain result abstain objective objective abstain abstain abstain abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method method abstain method abstain method abstain method abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method other other other other other method abstain abstain abstain abstain other abstain other abstain method abstain method method method abstain method method method abstain abstain other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method abstain method method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other abstain other other other other objective abstain objective abstain abstain result other other other