While a great deal of work has been done on NLP approaches to lexical semantic change detection, other aspects of language change have received less attention from the NLP community. In this paper, we address the detection of sound change through historical spelling. We propose that a sound change can be captured by comparing the relative distance through time between the distributions of the characters involved before and after the change has taken place. We model these distributions using PPMI character embeddings. We verify this hypothesis in synthetic data and then test the method's ability to trace the well-known historical change of lenition of plosives in Danish historical sources. We show that the models are able to identify several of the changes under consideration and to uncover meaningful contexts in which they appeared. The methodology has the potential to contribute to the study of open questions such as the relative chronology of sound shifts and their geographical distribution. The study of sound change goes back to the beginnings of modern linguistics in early nineteenth century, when scholars such as Rask and Grimm started making hypotheses about the way sound changes over time, which in turn lead to the discovery of regular sound correspondences between ancient languages and the identification of cognates in modern ones (Murray, 2015). Since spoken language from the past is not available, sound change in ancient languages must be deduced from written records by considering development in spelling through time. In addition, while we may be able to see from the written records that a change did occur, less is known on the specific dynamics of the change. Details of these dynamics include knowledge of when the change started to appear, how long it took for it to be complete, what was the relative chronology of individual sounds in a larger shift, what was the geographical distribution of a change and so forth. Due to the sparsity of linguistic evidence, detailed empirical studies of chronological sound change are difficult to conduct. This is especially the case for older stages of languages, where little written text was produced, and much of what did exist has been lost in transmission. However, as we move forward in history to the rise of bureaucracy, for example in medieval Europe, we see that an extensive amount of written records were made. Text from this period of time is interesting in the context of a study of sound change because it shows great variability in spelling patterns. With the digitalization of such archives 1 , therefore, new opportunities arise to apply computational methods to the study of sound change through written text. Considerable effort has already been devoted to the development of computational approaches aimed at discovering lexical semantic change (LSC) in historical corpora (Tahmasebi et al., 2018). However, change related to phonology, morphology and syntax has remained out of the spotlight in NLP research. In this study, we seek to bridge this gap as regards phonology. Inspired by the work on LSC, we propose a method whereby sound change is traced via spelling change in historical text and modeled by training diachronic character embeddings over text from different time periods. We start by reviewing previous approaches to the automatic detection of semantic shifts and spelling modification due to sound change. Then we formulate our hypothesis that a sound change can be traced using diachronic distributional embeddings. While sound change is not completely analogous to word meaning change, we argue that similar meth-1 A list of available resources for different languages is provided in the Guide to Medieval Manuscript Research from the University of Chicago Library: https://guides.lib. ods can be used for both. To verify our hypothesis, we conduct three studies on simulated sound change. First, we test the methods on the phonological environment of a simple artificial language. Then, we apply the same methods to a more complex scenario created by simulating sound change in a corpus of synchronic Danish text. Having established the suitability of the methods on these two datasets, we finally experiment with tracing a well-known sound change in real historical language data, again in Danish. The application of NLP methods to automatic LSC detection is already a rather well-developed sub-field of NLP research (Tahmasebi et al., 2018; Kutuzov et al., 2018). In particular, the emergence of word embeddings as a viable way to model the distributional hypothesis in semantics (Firth, 1957) has paved the way for an application of word embeddings to LSC modeling (Kim et al., 2014; Hamilton et al., 2016b; Eger and Mehler, 2016; Yao et al., 2018). Synchronically, the meaning of a word is characterized by word embeddings in terms of the contexts it appears in. LSC is captured by training word embeddings at different time points and comparing these distributions typically using cosine distance. Recently, contextualized word embeddings have also been applied to the problem. While such models have the capability to capture change in distinct word usages, preliminary results suggest that traditional word embeddings are su-perior to the task of semantic change detection (Schlechtweg et al., 2020; Montariol et al., 2021). One of the main issues in this comparison is the temporal alignment of dense embedding spaces. For example in the case of neural models, embeddings are initialized and trained stochastically, which means that separate runs  on even the same data  will yield different embedding spaces. Thus, work has focused on the development of methods to perform alignments to make dense embedding spaces comparable across time (see Kutuzov et al. (2018) for an overview). As an alternative to neural embeddings, scholars have also used purely count-based measures, which are naturally aligned across dimensions. Normalization techniques are 2 https://github.com/syssel/ letters-from-the-past also applied, e.g. based on positive pointwise mutual information (PPMI) (Hamilton et al., 2016b; Yao et al., 2018). Most studies of LSC do not rely on a control dataset against which to validate their conclusions. In Dubossarsky et al. (2017), on the contrary, it is argued that any claims about putative laws of semantic change in diachronic corpora must be evaluated against a relevant control condition. The authors propose a methodology in which a control condition is created artificially from the original diachronic text collection by reshuffling the data. No systematic LSC is expected in the artificially developed control dataset. The distributional hypothesis has also been proposed as an explanatory model within the domain of phonology suggesting that phonological classes are acquired through distributional information (Chomsky and Halle, 1968; Mielke, 2008). Driven by this hypothesis, recent work has focused on testing how distributional properties can be learned by phoneme embeddings (see Mayer 2020 for an overview). Silfverberg et al. (2018) investigated to what extent learned vector representations of phonemes align with their respective representations in a feature space in which dimensions are articulatory descriptors (e.g.,  plosive). Recently, Mayer (2020) has shown that phonological classes, such as long and short vowels, can be deduced from phoneme embeddings normalized using PPMI by iteratively performing PCA on candidate classes. Thus, while the distributional hypothesis for phonology is well-established, one notable issue is the fact that the empirical evidence to study sound change is relatively inaccessible since it requires recorded speech or phonologically transcribed data. Simulation is therefore used as a tool for studying the underlying mechanisms of sound change by creating computational models based on linguistic theory (Wedel, 2015). Through simulation, questions pertaining to e.g., what factors influence the (in)stability of vowel systems across generations (de Boer, 2003) can be modeled by controlling the assumptions made by the model. Work on simulation ranges from implementing theoretical approaches using mathematical models (Pierre-humbert, 2001; Blythe and Croft, 2012) to iterated learning and neural networks (Hare and Elman, 1995; Begu, 2021). level, they are primarily theoretically driven. In this paper, we wish to take a data-driven approach and utilize some of the methods reviewed above to track historical sound change in writing. Rather than using word embeddings as done to model lexical change, we will use character embeddings, that are better suited to the task of sound change modeling. Within the field of LSC detection, change in word semantics is traditionally measured by computing pairwise similarity (Hamilton et al., 2016b) over a time series, ( t , ..., t +  ), in which a shift in the meaning of a word, w i , can be measured by its relative distance to another word, w j . In this way, hypotheses about specific shifts may be tested. Another measure is semantic displacement , in which semantic change for a given word is quantified by measuring its temporal displacement. For both measures, looking at consecutive time steps provides a measure to the rate of change of a word  in relation to another word, or independently. While LSC is about meaning shifts of unchanged word forms, sound change is a change of form, i.e., a given phoneme changes to another one within certain contexts. We denote such a change a  b / c , where c' stands for a given context. While changes of either a or b will be reflected in changes to their individual distributions ( displacement ), looking at them independently of one another will not tell us whether one of the phonemes is becoming similar to the other. Therefore, we suggest to look at the pairwise similarity between a and b . More specifically, given a time series ( t 1 , ..., t n ), in which t 1 denotes a time before a sound change was in effect and t n denotes a time where a sound change is completed, we expect b i to move towards a 1 as i  n , in other words to become similar to a 1 , since it will begin to appear in the same contexts. As was noted earlier, sound is not accessible in historical text, to which we would like to be able to apply our methodology. In historical text preceding spelling conventions, sound is assumed to be reflected in spelling. While detailed philological and linguistics analyses of written language can help to determine if a distinct spelling corresponds to a particular phoneme, or whether that spelling is rather a reflection of synchronic spelling variation (Minkova, 2015), resources including such analyses are scarce. Thus, we chose to use characters as a proxy for sound, and model sound change through changes in the distance between pairs of character distributions. In addition, before assuming that an observed decrease in the distance between two such distributions reflects a real change, we also want to see that the same decrease is not visible in a control corpus in which no such change has indeed taken place. In order to verify the hypothesis that sound change can be traced using distributional information with the methodology proposed above, we test whether we are able to trace simulated change in synthetic data. As a first synthetic setting, we restrict ourselves to track change in a synthetic language with simple phonotactics. In this way, we get a sense of whether the proposed hypothesis works under perfect conditions, i.e., one in which characters correspond with phonemes one-to-one. In the second synthetic setting, we seek to imitate the condition of tracing change in an orthographic setting by simulating change in a corpus of synchronic text in which character distributions interact with the noise added by spelling and lexicon. In both synthetic settings, we compare the simulated change to a control setting where no change has occurred. Finally, we will test the hypothesis on real data. Our goal is to trace the lenition after vowels of voiceless plosives, p t k , to their voiced counterparts, b d g , in historical Danish. While this change is believed to have initiated around the beginning of the 14 th century, details about the relative chronology of the series and geographical distribution of the change are difficult to account for (Frederiksen, 2018). Therefore, in an attempt to discover interesting patterns of this change, we train character embeddings on historical sources from the periods following the time when the change is believed to have started. As we did for the synthetic data, and again following Dubossarsky et al. (2017), we also introduce a control setting to test the significance of the observed changes. Parupa is an artificial language introduced by Mayer (2020). It is characterized by a small phonological inventory 3 , and simple phonotactic rules for how sounds combine:  only CV syllables are allowed 3 C : / p t k b d g r / V : / i e u o a / 6715  / p t k / occur before high vowels, / i u /  / b d g / occur before non-high vowels, / e o /  only / b p / occur word-initially  / r / occurs before all vowels  all consonants can occur before / a / We created five corpora of Parupa consisting of 20,000 words each using the Hidden Markov Model provided by Mayer (2020) 4 : While the first corpus, parupa 1 , preserves the phonotactic rules listed above, the remaining four include a sound change, p  b /_ u, i 5 , which happens gradually (linearly) and is fully completed in parupa 5 . Additionally, we created five control corpora (one for each of the target ones and with the same vocabulary) which do not include any simulated sound change. Each of the corpora consists of 50 , 000 words. The Danish UD treebank To collect a corpus of synchronic language, we extracted the training sentences from the Danish UD treebank (Johannsen et al., 2015). From this collection of sentences, we extracted five sub-corpora ( UD-Danish 1-5 ) consisting of  16,000 words each, in which we simulated a sound change, g  k / V_{V # t#} 6 . As done in the case of Parupa, the sound change was simulated gradually, with linear increase in change probabilities (i.e., 0 , 0 . 25 , 0 . 50 , 0 . 75 , 1 ). To create the control condition, we also kept a version of the sub-corpora where no change was simulated. The five control versions are thus identical to the five target corpora in terms of vocabulary and distributions, except for the simulated change. Historical spellings of geographical names Danmarks Stednavne is a on-going lexicographic book series creating a register of geographical names in Denmark. The register also serves as a philological resource by listing attestations of the names coming from various historical resources. distributional_learning 5 The underscore indicates the position of the changing element, i.e., p changes into b when preceding u or i . This notation using an underscore to indicate position of the changing element will be used throughout the rest of the paper. 6 i.e., g between vowels, word-final after vowel, or after vowel preceding word-final t . For example, the entry for Copenhagen includes over 700 historical attestations listed by date 7 . In addition to the printed volumes (Danmarks Stednavne, 19222013), geographical names and their connected metadata (e.g., geographical location and historical attestations) have been digitized, and can be found in an online edition 8 which comprizes over 210 , 000 names and 900 , 000 historical attestations. To study the lenition of / p t k /, we extracted historical attestations of names ranging from the 12 th to the 18 th century. Using the attestation before the 14 th century as a reference to the time before the change was initiated ( t 1 ), we divided the list of names into bins of half a century to track the development of character embeddings through time. The choice of bin size is an important methodological consideration when tracing language change (Kutu-zov et al., 2018). From a philological perspective, 50 years correspond to two generations of writers (spellers'), which is considered a realistic bin size to track development of spelling in writing. This provides us with eleven sub-corpora with 31 , 000 (  15 , 000 ) name tokens on average. In order to create a control setting, we generated a corresponding number of sub-corpora by stratifying the names with respect to their date of attestation, corresponding to the shuffle' approach suggested by Dubossarsky et al. (2017). In this approach, no diachronic change is expected to be observed, as attestations are distributed evenly across bins based on their original date of occurrence. To represent characters in a distributional space, we use PPMI embeddings. Contrary to dense embeddings, these are easy to interpret and when compared across different initializations, they are naturally aligned, so we do not introduce noise caused by the alignment process. Using the implementation by Mayer (2020), the sliding window is directional, and thus we distinguish contexts preceding and following the target character. While this directionality is neglected when creating PPMI word embeddings, the direction matters when using character embeddings to test the intuition behind the distributional hypothesis, in which direction in a context is meaningful. conditioning of the change aimed to be captured: For Parupa, the simulated change is conditioned on only one character, and thus for this experiment we applied bigrams. For UDDanish, we applied trigams as the change is conditioned by two characters (the preceding and succeeding). For the tracking of lenition in Danish, the condition of the change is expected to be similar to the one we simulated in the synthetic setting of UDDanish. However, to ensure we capture enough context, in this case we expand the model to using 4-grams. We measure sound change in terms of a decrease in the distance between two character distributions over time. In other words, given two character distributions A and B corresponding to any two phonemes / a / and / b /, we should see that distance ( A (1) , B ( n ) ) gets smaller for greater values of n if there is a change A  B . While most studies use cosine distance to measure the difference between distributions (Kutuzov et al., 2018), we chose to use Euclidean distance as it directly reflects our hypothesis by taking the sum of differences in each dimension (context). For each of the corpora being investigated, we use the R software (R Core Team, 2021) and the effects' package (Fox and Weisberg, 2019) to build linear regression models that predict the distributional distance between two sounds per temporal interval in the target and the control versions of the corpus. The advantage of employing linear regression in this case is that we can test the effect of multiple factors as well as their interaction. In our case, the distance between the two sounds being investigated is the dependent variable, and we want to predict the main effects of temporal interval and corpus as well as the interaction between them. To argue that there has been a sound change across time, there must be a significant effect of temporal interval on distance. In addition, we would like to see an interaction between this effect and the effect of the corpus variable in that the change should be absent, or at least significantly smaller, in the control corpus. Table 1 shows the results of the linear regression models we developed to test whether any evidence of sound change discovered in the target corpora, where sound change is either simulated or histori-3 The intercept' estimate corresponds to the distance predicted between the two sounds being investigated in the initial temporal interval. The Bin' estimate shows by how much the distance is expected to change for every temporal interval. A negative effect means that the distance between the two sounds is becoming smaller. The Con-trol' effect shows the predicted change to the initial Intercept in the control corpus (this corresponds to the effect of the corpus variable), and finally Bin:Control' shows the interaction between temporal bin and corpus type. In both corpora where change is simulated, there is a significant effect of temporal interval. This is expected given the fact that gradual change has been induced in the data. For both corpora, the effect of the control corpus on the initial sound distance is not significant. Importantly, the interaction between the effect of the temporal bin and the control corpus is significant in both cases. The 6717 Effect Estimate Std. interaction supports the hypothesis that we see a pattern of change in the simulated corpora that is significantly different compared to the control data. The interactions are shown in the plots in Figure 1. Turning to the results for the Danish Geographical Names corpus, while the models show significant effects of Bin, Control and interaction between the two for the k  g and the t  d changes, no significant effects are found for the p  b change. When we look at the corresponding interaction plots in Figure 2, we see that the distance between p and b in the corpus decreases in the third bin to then increase and finally slightly decrease again in a non-linear way. The changes displayed in the plots in (b) and (c), on the contrary, follow the expected trend: The observed consonant is moving towards its voiced version in the real corpus but not in the control. The results from the two simulation studies suggest that sound change can be traced with our proposed methodology of measuring the distance between pairs of character distributions over time. We showed this both in a simplified setting (Parupa), and in the orthographically noisy environment provided by synchronic Danish data (UD Danish). The main assumption in these simulation studies was that change could be modeled linearly. However, as discussed by scholars, change is often not linear, and can follow an s-shaped curve through a community (Denison, 2003). In a study of semantic lexical change based on synthetic data, Shoe-mark et al. (2019) experiment with the injection of changes the probabilities of which vary linearly or logarithmically, and find that regression in general provides reasonable results in spite of being sensitive to outliers and of producing a certain amount of false positive results. It also performs better than a non-parametric measure like Kendall's  . The results obtained in our study seem to confirm the usefulness of linear models to detect sound change even though one of the cases of lenition targeted in the Danish Geographical Names corpus could not be modelled. Focusing on our results on the tracing of lenition, then, we were able to identify a change from /t k/  /d g/ . However, an important thing to note in regards to the control setting for these results is how it diverges from the synthetic settings, which we initially used as a verification of the proposed hypothesis to trace sound change. There, the 6718 102 103 104 105 Interaction of Bin and Corpus on Distance in Geo: p => b D i s t an c e  1300 1300  1350 1350  1400 1400  1450 1450  1500 1500  1550 1550  1600 1600  1650 1650  1700 1700  1750 1750  1800 Corpus ChangeControl variation in vocabulary was the same in the simulated and the control settings. In this case, however, vocabulary variation is lower in our control setting due to the shuffling of the name attestations. As a consequence, the control setting does not properly test the possible confounding effect of vocabulary within the proposed methodology. Therefore, we proceeded to evaluate what types of contexts the model picked up. To get a sense of this, instead of looking at the euclidean distance for the full embedding, we ran linear regression on the target data looking at differences between character distributions for each dimension. We then extracted the patterns corresponding to the dimensions showing significant differences and considered those with the highest Pearson's r coefficient (Tables 2-4). Starting with the resulting patterns for Parupa and UD Danish, in both cases we are able to identify the exact contexts where the change was simulated: In Parupa before i / u and in the UD Danish corpus, between vowels and in the frequent suffix ig(t) (although the end-of-word is not captured due to n-gram size restrictions). For Parupa, it is worth noting how the model captures patterns after vowel as well. This position is only implicitly involved in the conditioning of the simulated change, and the 6719 4-gram Slope Pearson's r rvi_ -0.49 -0.85 _er -0.42 -0.78 sii_ -0.40 -0.71 m#a_ -0.40 -0.81 oli_ -0.39 -0.84 an_h -0.32 -0.80 ara_ -0.31 -0.62 n_ga -0.29 -0.82 vi_# -0.29 -0.70 is_a -0.29 -0.73 Table 4: Analysis of the change from k to g in historical records of geographical names. slope correspondingly less steep. Moving on to the tracing of sound change in real data, we focus our analysis on k  g , which showed the greatest change. Considering the patterns, rvi_ and vi_# , these are connected to the the word vig inlet', commonly used as a suffix in the formation of geographical names in Danish. Descending from a Proto-Germanic word with final k ( wkwan to give way; to turn (away)', compare German weichen id.' and Dutch wijken id.' (Kroonen, 2013)), the suffix is in early sources attested with a k : For example, out of the six written sources of the geographical name Rrvig before the 14 th century (corresponding to bin 1-3 in our study), four were written with a k , while in later sources forms with g became predominant, with the latest attestation of k appearing in 1465. Many of the patterns can be attributed to spellings related to similar changes 9,10 . However, in the case of n_ga , is_a and an_h these are not immediately interpretable. In the case of oli_ , this pattern is most likely related to the word bolig home;dwelling'. This word, however, does not have a comparable ancestor with k , and the change has to be explained as reflecting later innovation, namely beginning trend of using bolig in name formations among younger attestations. This latter example is related to an important issue in language evolution: When language changes 9 Danish sig bog; mire' from Old Danish sik , compare Norwegian and Swedish (dialectal) sik (Danmarks Stednavne, 19222013) 10 Danish ager field' from Proto-Germanic akra , compare English acre and Swedish ker (Kroonen, 2013). through generations, we also observe shifts in culture. Different types of data drift' are in fact discussed by Hamilton et al. (2016a) in the context of LSC. The authors suggest that they may be modeled independently of each other by means of different measures of change. The effect of cultural change has yet to be discussed for sound change. However, it is an important discussion, since phonology, when looking at it from a corpus-based perspective, is not only governed by phonotactic constraints, but also a by-product of word usage, which is in turn dependent on cultural patterns. In this respect, another important point to note about the retrieved patterns  both from the simulation of UD Danish and the tracing of k  g  is that many of them reflect derivational or inflec-tional suffixes, and are thus characterized by high frequency of occurrence across word forms. While the observation that frequent patterns are more easily captured may seem trivial, lack of suffi-cient evidence may nevertheless be the reason why we cannot model the p  b change. Germanic p descends from Proto-Indo-European (PIE) * b , which, however, has a special place in the PIE phoneme inventory and is considered a sort of black sheep that some scholars do not believe to have existed due to its few attestations. In fact, the attestations of Germanic p most often come from loan words and are not seen in morphemes. Thus the evidence for p  b is inherently scarcer than for the other two consonant pairs we have investigated. Further investigation of this sound change could be carried out by means of additional simulations, or more detailed analysis of the obtained character distribution, and is left for the future. A final observation on the identified patterns is that the model fails to generalize across synchronic variation in spellings. For example, we see that a spelling with ii is treated alongside spelling with a single i . While this type of variation could to some extent be accounted for by treating it as an independent variable, such a solution would have consequences for our experiment design in that we use PPMI weighting on raw n-gram counts. This method enabled us to interpret the exact inner workings of the model and find the contexts in which a change has happened. If we had used neural models for example, in which characters are represented by dense embeddings, similar characters would have shared similar representations, thereby 6720 perhaps allowing the model to generalise e.g., to sound change occurring after a vowel . In this study, we wanted to privilege explainability, but dense representations should be explored in the future. In this paper we presented a novel method for the modeling of sound change through the use of diachronic character embeddings. Sound change is modeled in terms of increasing similarity between character distributions across time intervals. The proposed method was tested on synthetic data with promising results, and then applied to a real world scenario with the goal of tracing the lenition of / p t k /  / b d g / in Danish by looking at spelling in historical sources. The method was able to detect the changes for two of the sound pairs, and also to point at specific contexts of occurrence that influenced the changes. However, our evaluation showed that the proposed models were sensitive to variation relating to vocabulary. To what extent such variation is responsible for the occurrence of false positives has yet to be investigated. For scholars interested in sound change, there are a number of important open questions, such as the relative chronology and geographical distribution of sound shifts. Although we have not addressed these questions here, we believe our methodology can be further developed in ways that would allow to do so, e.g., by adding geographical location as an additional factor in the models. Both issues would constitute interesting avenues for future research. In this paper we have used purely count-based methods. While this approach enables us to directly interpret the results of the models, it also suffers from its inability to generalise across contexts. This drawback motivates experimenting with neural methods that make use of dense character representations, to test whether they can make similar generalisations as done by historical linguists, particularly as regards infrequent patterns that could be captured across word forms. We would like to thank Bo Nissen Knudsen and David Caspersen Yousif for helping to prepare and making the data from Danmarks Stednavne available for the study. Also, a great thanks to the anonymous reviewers for their helpful comments and suggestions. and Text in Time and Space , a core group project funded by the Velux Foundations.
abstain abstain objective method result objective abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain objective abstain method other abstain result objective method method other other other other other other other other other other other other other other other other other other other other other other other other method method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain method method method method method method method abstain method method abstain method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method method abstain method method method abstain method method method method method method method method method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain method abstain result result abstain other abstain result abstain method abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method result method abstain objective abstain abstain abstain objective abstain abstain objective abstain method result abstain other other other