Human innovation in language, such as inventing new words, is a challenge for pretrained language models. We assess the ability of one large model, GPT-3, to process new words and decide on their meaning. We create a set of nonce words and prompt GPT-3 to generate their dictionary definitions. We find GPT-3 produces plausible definitions that align with human judgments. Moreover, GPT-3's definitions are sometimes preferred to those invented by humans, signaling its intriguing ability not just to adapt, but to add to the evolving vocabulary of the English language. Humans are constantly expanding languages with new words. How are artificial language models, which are increasingly deployed in the wild', to handle the stream of neologisms that are appearing in slang or on social media (Grieve et al., 2018)? Today's most advanced language models, including GPT-3 (Brown et al., 2020), use a subword tokenization of input text, rather than consuming it word by word. This allows them to process words never seen in their training data. For example, the word perdetry', which has never been used in English, is treated by GPT-3 as a sequence of two tokens (Fig. 1). The subword tokenization algorithm is designed for text compression and does not respect the natural morpheme boundaries. We explore GPT-3's understanding of English at the subword level by prompting it to give definitions of nonce words 1 (Fig. 1). We find in human 1 We use the term nonce word' for a new word not used in English. per|detry (n.) an instance of inventing words, esp. as a hobby har|bole|mic (adj.) tending to babble; talking nonsense sh|out|ze (v.) to laugh through half-open teeth studies that not only does GPT-3 generate realistic, original meanings for new words, but its definitions are sometimes preferred to those invented by humans. This finding sheds light on GPT-3's ability to adapt to and even extend a changing vocabulary. While we cannot ascertain GPT-3's exact reasons for assigning meanings to nonce words, our results prove that these reasons are not limited to morphology: many neologisms have no clear roots or derivational origin. The meanings of words may be imported by their phonological qualities  more precisely, their orthographic realizations  or by clues to their membership in certain lexical strata. Thus, at a high level, our findings suggest that GPT-3 has learned not only its world knowledge and capacity for long-range reasoning in text (Brown et al., 2020), but also the nuances of etymology and the correspondences of sound and meaning that lie at the very base of language understanding. Below are some pairs of words together with their definitions. The goal is to guess, for each pair, which word goes with which definition. We will show you two options, and you will decide which of them is a better match. The words you'll get are rare, and we do not expect you to know many, or indeed any, of them. Make your best guess. For some pairs, there is no correct answer. We'll show you the expected answers at the end. Do not look up the words while doing the task: we are really interested in your gut feeling, right or wrong. In his seminal work, de Saussure (1916) rejected this notion. Yet, later work identified a large set of English phonesthemes, such as the cluster /gl/ in glow', glitter', gloss', etc. meaning light; a notable list was compiled by Marchand (1959a,b). Recent studies found phonosemantic patterns that are common to many languages (Blasi et al., 2016). In practice, words are even engineered for subconscious reactions: certain sounds in brand names are correlated with associations such as size (of a gadget) or speed (of a courier) (Klink, 2000). Our study suggests that GPT-3 may understand such patterns as well. There is a body of work on joint modeling of (or-thographic or phonological) word forms and grammatical classes such as noun gender and inflection pattern. In a recent study, Williams et al. (2020) used neural models to measure mutual information between meanings and inflection classes of Czech and German nouns, which, for borrowed words, often depend on the language of origin. It is plausible that GPT-3 implicitly uses likely source languages of nonce words to generate meanings associated with some lexical strata, e.g., abstract nouns from Norman French, concrete nouns from the Germanic substrate, and artificially constructed terms with Greek or Latinate elements. (We direct the interested reader to the lexicon in Appendix C.) them into embedding spaces (Bojanowski et al., 2017; Zalmout et al., 2019; Ryskina et al., 2020), and codifying and predicting etymologies (Melo, 2014; Wu and Yarowsky, 2020). Others have studied definition generation (Noraset et al., 2017) and the reverse task of mapping definitions to words (Hill et al., 2015), albeit with pretrained embeddings. Limited examples of a pretrained model's use of nonce words appear in Brown et al. (2020). In this work, we study GPT-3's ability to define words never seen in context. We trained a LSTM model (Hochreiter and Schmid-huber, 1997) on a corpus of English words 2 with a standard character-level objective, then sampled strings from the LSTM to create nonce words. The words were lemmatized and assigned parts of speech (POS): noun (n.), verb (v.), or adjective (adj.). To produce definitions for these words, we generated text from GPT-3, primed with input in the format  word ( POS. ) . Usually, GPT-3's outputs had the style of a dictionary definition (Fig. 1). The definitions were filtered by common-sense criteria and lightly edited for consistency, as explained in Appendix A. By this procedure, we obtained 146 word-definition pairs (67 n., 47 v., 32 adj.). For comparison in our study, we also sampled a set of real but rare English words from a corpus. 2 Definitions for these words were taken from a dictionary. 3 This resulted in a combined set of 220 words (102 n., 70 v., 48 adj.), with a 2:1 ratio of fake to rare words in each POS. See Appen-2 github.com/dwyl/english-words 3 en.wiktionary.org n. dices A and C for the full lexicon and generation details, including all points of human input. We performed a study in which human subjects were presented with pairs of words of the same part of speech together with their definitions (generated by GPT-3, for fake words, or extracted from the dictionary, for rare words), but not told which definition matches with which word. 4 Some questions contained two fake words, some two rare words, and some one fake and one rare word. Users were asked to decide which assignment of definitions to words is a better fit and to rate their confidence (Fig. 2); the choices were converted to a scale of 0 (confident in the incorrect match) to 5 (confident in the correct match). Each user received a random pairing of words, but saw each word exactly once. We collected 65 sets of annotations for each POS, for a total of 65  2202 = 7150 data points. Results. Humans prefer the pairing from our lexicon in 68% of cases. The scores by the POS and the kind of pair (fake-fake, fake-rare, or rare-rare) are shown in the top rows of Table 1. GPT-3's definitions align with human judgments far better than random choice ( p -values below floating-point epsilon). Notably, humans' performance on pairs containing a fake and a rare word was about the same as on pairs of fake words. Correlation in performance between different parts of the word-definition matching task is high. Considering only the fake-fake pairs, the score (number of correctly matched pairs) on the noun portion of the task is correlated with the score on 4 Users were not told that some of the definitions were machine-generated. verb and adjective pairs with Spearman   0 . 42; a permutation test on rank correlation gives p  0 . 01. The verb and adjective portions are similarly predictive of the other two ( p  0 . 05 for both). The correlation is even stronger ( p < 0 . 0001 for nouns) when all pairs, not just fake-fake, are considered. This indicates that some users can be identified as better' at the task, perhaps due to their personal vocabulary, education, or effort. (For example, the average score on the fake-fake noun pairs is 70.7%. However, the average score on fake-fake noun pairs among users who scored above median on the fake-fake adjective pairs is 74.2%.) This is strong evidence that the values in Table 1 would be higher with a better selection of users. There was significant agreement between annotators. In cases when the same pair of words was shown to two users, the mean difference between the two users' choices on the 0-5 scale was 1.5, and in 61% of cases the two users preferred the same assignment. Remarkably, the latter number is the same for rare-rare, rare-fake, and fake-fake pairs. It is possible that the subjects knew some of the rare words  and the tables in Appendix C do suggest this. However, assuming that a subject will choose the correct match if they know the meaning of at least one word in a pair, and will do no worse than random guessing on pairs where they know neither word, the last human' row is consistent with less than a quarter of the rare words, on average, being known to the subjects. Likelihood analysis. For each word w and definition d in the lexicon (where d may be the definition of a word different from w ), we compute the likelihood under GPT-3 of the definition d to follow word w , p ( d | w ) . For each pair of words ( w 1 , w 2 ) of the same POS, with definitions ( d 1 , d 2 ) , we compute the difference in log-likelihood between the proper match ( w 1  d 1 , w 2  d 2 ) and the inverted assignment ( w 1  d 2 , w 2  d 1 ) : LLD ( w 1 , w 2 ) = log p ( d 2 | w 1 ) p ( d 1 | w 2 ) p ( d 1 | w 1 ) p ( d 2 | w 2 ) . If GPT-3 were to perform the matching task done by our human subjects, it would choose the option with higher total likelihood. In other words, it would prefer the correct pairing if LLD ( w 1 , w 2 ) is negative and the inverted pairing if it is positive. the correct matches for fake-fake pairs, since the definitions of fake words were sampled from the same model of likelihood. Indeed, we see this in the bottom rows of Table 1. GPT-3's imperfect performance on fake-fake pairs is a byproduct of the sampling used in the generation and perhaps of the edits made in postprocessing. To maximize total likelihood of the lexicon, GPT-3 would prefer to enact some post-factum swaps of definitions. LLD and human confidence. LLD is a good predictor of human judgments: confidence in the correct pairing for fake-fake pairs ( w 1 , w 2 ) is strongly correlated with LLD ( w 1 , w 2 ) , a rank correlation test giving p < 0 . 001 for all POS. One may object that this correlation  and indeed much of humans' performance  is due to the presence of simple disambiguating markers: for example, a word with suffix -ist' is likely to denote a person, while an -ism' is probably an abstract noun. However, examination of log-likelihood differences shows that this is not the case. We stratify the pairs of fake words by LLD and consider the distribution of humans' confi-dences for pairs with LLD falling in five ranges: [  40 ,  30 ) , [  30 ,  20 ) ,..., [ 0 , 10 ) . Confidence in the correct matching is inversely correlated with LLD, but humans tend to choose the correct assignment for pairs in all five strata (Table 2). For pairs with LLD in the ranges [  10 , 0 ) and [ 0 , 10 ) , which form a majority, there tend to be no revealing morphological markers. (Table 2 shows pairs of words with LLD falling into these ranges; Table 7 in the appendix shows more examples.) Conclusion. Finally, we observe that many of GPT-3's definitions are original: we are not aware of English words that describe the same concepts (see Table 2 and Appendix C). Some of the innovated meanings fill plausible lexical gaps (drob-bler'), while others require a degree of creativity (subacitide'). This shows that GPT-3 is not simply aligning new words with existing words as in Zalmout et al. (2019), but inventing new meanings. We test GPT-3's ability to define new words on a set of human-proposed neologisms from the Dictionary of Obscure Sorrows. 5 Many of these words were created out of real English morphemes. We sampled 20 words from this set, got GPT-3 defini-5 dictionaryofobscuresorrows.com/ A. occhiolism : a belief that personal power increases proportionally with one's height B. occhiolism : the awareness of the smallness of one's perspective Figure 3: A typical question in the definition choice task. We then ran a study with 25 users, in which each user was given words and both definitions (in random order, without being told how each definition was generated) and asked to pick the better match. The responses were converted to a scale of 0 (human-generated is much better) to 5 (GPT-3-generated is much better). Each user marked their definition preference for all 20 words (Fig. 3). Results. Remarkably, users preferred GPT-3's definitions in 40% of cases, despite the fact that a human thought up each of these word-meaning pairs. This is not simply the result of random guessing by the workers: the result matrix (Fig. 4) shows a significant amount of structure. There are words on which most users agree that the better definition is the one generated either by the human inventor (top rows) or by GPT-3 (bottom rows). Users most prefer GPT-3's definition for backmasking : the act of disguising messages within recordings via sound effects to the human definition the instinctive tendency to see someone as you knew them in their youth , while the human definition of lapyear : the age at which you become older than your parents were when you were born is preferred to GPT-3's a lazy person; someone of a low-energy lifestyle . User clusters. These human-coined neologisms have a bias towards meanings with an existential slant, which results in additional structure in our results, reflecting the population structure of the subjects. Indeed, some workers prefer human-made definitions and others prefer GPT-3's definitions, which reflect a mixture of meanings seen in a crawl of the Internet. To analyze the significance such preferences, we perform a randomization test. We define the polarization of a user as the absolute difference between the number of words for which they prefer the human-generated definition and the number for which they prefer GPT-3's definition. The average polarization over users is greater than that seen in 99% of random preference matrices, indicating that there may indeed be two types of users, with different preferences for the types of meanings they see in words. 6 5 Conclusion A character-level model of English words composed with GPT-3 is a complete scheme for generating new words and innovative meanings. GPT-3 invents definitions for words it has not seen in training that are seen as reasonable by humans. These results have implications for language models' ability to adapt and even add to an evolving vocabulary. They can inspire future work on machine understanding of new slang, optimization of words and acronyms, creation of fictitious entries, and automatically generating word games. 6 A similar test could be performed taking the confidence into account. Here we define polarization as the absolute difference between a user's mean confidence and 2.5. In each random sample, we flip a random subset of the entries in the confidence matrix to the opposite preference, while keeping the level of uncertainty the same: 0  5, 1  4, 2  3. This results in a p -value around 0.04. immediate negative societal this work. Jack Grieve, Andrea Nini, and Diansheng Guo. 2018. Mapping lexical innovation on American social media. Journal of English Linguistics , 46(4):293319. Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics , 4. Sepp Hochreiter and Jrgen Schmidhuber. 1997. Neural Computation , 9(8):1735-1780. Creating brand names with meaning: The use of sound symbolism. Marketing Letters , 11:520. Phonetic symbolism in English word-formation. Indogermanische Forschun-gen , 64:146168. Janis Nuckolls. 1999. The case for sound symbolism. Annual Review of Anthropology , 28:225252. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP) , pages 15321543. As explained in Appendix B, we followed data privacy and anonymization procedures to the greatest extent possible and fairly compensated human subjects.
abstain objective method result abstain abstain abstain abstain abstain abstain abstain objective objective abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain method abstain abstain method abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method