There is no consensus on the state-of-the-art approach to historical text normalization. Many techniques have been proposed, including rule-based methods, distance metrics, character-based statistical machine translation, and neural encoderdecoder models, but studies have used different datasets, different evaluation methods, and have come to different conclusions. This paper presents the largest study of historical text normalization done so far. We critically survey the existing literature and report experiments on eight languages, comparing systems spanning all categories of proposed normalization techniques, analysing the effect of training data quantity, and using different evaluation methods. The datasets and scripts are made publicly available. Spelling variation is one of the key challenges for NLP on historical texts, affecting the performance of tools such as part-of-speech taggers or parsers and complicating users' search queries on a corpus. Normalization is often proposed as a solution; it is commonly defined as the mapping of historical variant spellings to a single, contemporary normal form as exemplified in Figure 1. Automatic normalization of historical texts has a long history, going back to at least Fix (1980). Earlier approaches often rely on hand-crafted algorithms tailored to one specific language, while more recent approaches have focused on supervised machine learning, particularly character-based statistical machine translation (SMT) and its neural equivalent (NMT). However, no clear consensus has emerged about the state of the art for this task, with papers either reporting an advantage for NMT (Hmlinen et al., 2018), SMT 1 This work largely builds upon the author's doctoral thesis (Bollmann, 2018), the research for which was carried out at Ruhr-Universitt Bochum, Germany. (Domingo and Casacuberta, 2018), or language-specific algorithms (Schneider et al., 2017). Moreover, the quantity of annotated training data varies considerably between studies, making it diffi-cult to obtain practical recommendations for new projects seeking to use normalization techniques. Contributions This paper aims to provide the most comprehensive evaluation and analysis of historical text normalization systems so far. Motivated by a systematic review of previous work on this topic (Sec. 2), only publicly available normalization systems covering a wide range of proposed techniques are selected (Sec. 3) and evaluated across a diverse collection of historical datasets covering eight languages (Sec. 4). This is followed by a detailed analysis of the effect of training data quantity and a critical discussion of evaluation methods for assessing normalization quality (Sec. 5). The datasets and code are made freely available whenever possible, 2 along with detailed instructions on how to reproduce the experiments. 2 https://github.com/coastalcph/ histnorm ; one dataset could not be included due to licensing restrictions. The following overview is broadly organized by categories that each represent a conceptually or methodically different approach. The conceptually simplest form of normalization is to look up each historical variant in a precompiled list that maps it to its intended normalization. This approach can go by many names, such as lexical substitution , dictionary lookup , wordlist mapping , or memorization . While it does not generalize in any way to variants that are not covered by the list, it has proven highly effective as a component in several normalization systems, such as the semi-automatic VARD tool (Rayson et al., 2005; Baron and Rayson, 2008) or the fully automatic Norma tool (Bollmann, 2012). Rule-based approaches try to encode regularities in spelling variantse.g., historical (cid:104) v (cid:105) often representing modern (cid:104) u (cid:105) in the form of replacement rules, typically including context information to discriminate between different usages of a character. Some of the earliest approaches to normalization are rule-based, with rules being created manually for one particular language, such as Old Icelandic (Fix, 1980) or Old German (Koller, 1983). VARD 2 uses letter replacement rules to construct normalization candidates, but is not necessarily concerned with precision due to its interactive nature (Baron and Rayson, 2008). Bollmann et al. (2011) describe a supervised learning algorithm to automatically derive context-aware replacement rules from training data, including identity rules that leave a character unchanged, then apply one rule to each character of a historical word form to produce a normalization. Porta et al. (2013) model phonological sound change rules for Old Spanish using finite-state transducers; Etxeberria et al. (2016) describe a similarly motivated model that can be trained in a supervised manner. Rule-based methods are also commonly found when the goal is not to produce a single best normalization, but to cluster a group of spelling variants (Giusti et al., 2007) or to retrieve occurrences of variant spellings given a modern form in an information retrieval (IR) scenario (Ernst-Gerlach and Fuhr, 2006; Koolen et al., 2006). Approaches using edit distance measures (such as Levenshtein distance; Levenshtein, 1966) are most commonly found in an IR context, since measures that compare two word forms are a natural fit for matching a search term with relevant word forms in a historical document (e.g., Robertson and Willett, 1993). Weighted variants of distance measures can be used to assign lower costs to more likely edit operations (Kempken et al., 2006; Hauser and Schulz, 2007). In a normalization context, distance measures can be used to compare historical variants to entries in a contemporary full-form lexicon (Keste-mont et al., 2010; Jurish, 2010a). Norma includes a distance-based component whose edit weights can be learned from a training set of normalizations (Bollmann, 2012). Pettersson et al. (2013a) find a similar approach to be more effective than hand-crafted rules on Swedish. Sometimes, the line between distance-based and rule-based methods get blurred; Adesam et al. (2012) use the Levenshtein algorithm to derive substitution rules from training data, which are then used to link up historical Swedish forms with lexicon entries; van Halteren and Rem (2013) describe a comparable approach for Dutch. Furthermore, distance measures also lend themselves to unsupervised approaches for clustering historical variants of the same modern form, where identifying the precise modern form is not necessarily required (Amoia and Martnez, 2013; Barteld et al., 2015). In a probabilistic view of the normalization task, the goal is to optimize the probability p ( t | s ) that a contemporary word form t is the normalization of a historical word form s . This can be seen as a noisy channel model , which has been used for normalization by, e.g., Oravecz et al. (2010) and Etxeberria et al. (2016). More commonly, character-based statistical machine translation (CSMT) has been applied to the normalization task. Instead of translating a sentence as a sequence of tokens, these approaches translate a historical word form as a sequence of characters. This has been found to be very effective for a variety of historical languages, such as Spanish (Snchez-Martnez et al., 2013), Icelandic and Swedish (Pettersson et al., 2013b), Slovene (Scherrer and Erjavec, 2013, 2016; Ljubeic et al., 2016), as well as Hungarian, German, and English (Pettersson, 2016), where it is usually found to outperform previous approaches. Pettersson et al. (2014) find that a CSMT system often performs best in a comparison with a filtering method and a distance-based approach on five different languages. Schneider et al. (2017) compare VARD 2 to CSMT on English and find that VARD 2 performs slightly better. Domingo and Casacuberta (2018) evaluate both word-based and character-based models and find that SMT outperforms a neural network model. Neural network architectures have become popular for a variety of NLP tasks, and historical normalization is no exception. Character-based neural machine translation (CNMT) is the logical neural equivalent to the CSMT approach, and has first been used for normalization of historical German (Bollmann et al., 2017; Korchagina, 2017) using encoderdecoder models with long short-term memory (LSTM) units. Robertson and Goldwater (2018) present a more detailed evaluation of this architecture on five different languages. Hmlinen et al. (2018) evaluate SMT, NMT, an edit-distance approach, and a rule-based finite-state transducer, and advocate for a combination of these approaches to make use of their individual strengths; however, they restrict their evaluation to English. Other neural architectures have rarely been used for normalization so far. Al Azawi et al. (2013) and Bollmann and Sgaard (2016) frame the normalization task as a sequence labelling problem, labelling each character in the historical word form with its normalized equivalent. Keste-mont et al. (2016) use convolutional networks for lemmatization of historical Dutch. Overall, though, the encoderdecoder model with recurrent layers is the dominant approach. The presented literature almost exclusively focuses on models where the input is a single token. In theory, it would be desirable to include context from the surrounding tokens, as some historical spellings can have more than one modern equivalent depending on the context in which they are used (e.g., historical ther could represent their or there ). Remarkably few studies have attempted this so far: Jurish (2010b) uses hidden Markov models to select between normalization candidates; Mitankin et al. (2014) use a language model in a similar vein; Ljubeic et al. (2016) experiment with segment-level input, i.e., a string of several historical tokens as input to a normalizer. Since this area is currently very underex-plored, it warrants a deeper investigation that goes beyond the scope of this paper. Systems The selection of normalization systems follows two goals: (i) to include at least one system for each major category as identified in Sec. 2; and (ii) to use only freely available tools in order to facilitate reproduction and application of the described methods. To that effect, this study compares the following approaches:  Norma 3 (Bollmann, 2012), which combines substitution lists, a rule-based normalizer, and a distance-based algorithm, with the option of running them separately or combined. Importantly, it implements supervised learning algorithms for all of these components and is not restricted to a particular language. cSMTiser 4 (Ljubeic et al., 2016; Scherrer and Ljubeic, 2016), which implements a normalization pipeline using character-based statistical machine translation (CSMT) using the Moses toolkit (Koehn et al., 2007). Neural machine translation (NMT) , in the form of two publicly available implementations: (i) the model by Bollmann (2018), also used in Bollmann et al. (2018); 5 and (ii) the model by Tang et al. (2018). 6 Two systems were chosen for the NMT approach as they use very different hyperparameters, despite both using comparable neural encoder decoder models: Bollmann (2018) uses a single LSTM layer with dimensionality 300 in the encoder and decoder, while Tang et al. (2018) use six vanilla RNN cells with dimensionality 1024. 3 https://github.com/comphist/norma 4 https://github.com/clarinsi/csmtiser 5 I reimplemented the model here using the XNMT toolkit (Neubig et al., 2018). 6 https://github.com/tanggongbo/ normalization-NMT ; their model uses the deep transition architecture of Sennrich et al. (2017, Sec.2.3.1) as implemented by Marian (Junczys-Dowmunt et al., 2018). Datasets Table 1 gives an overview of the historical datasets. They are taken from Bollmann (2018) and represent the largest and most varied collection of datasets used for historical text normalization so far, covering eight languages from different language familiesEnglish, German, Hungarian, Icelandic, Spanish, Portuguese, Slovene, and Swedishas well as different text genres and time periods. Furthermore, most of these have also been used in previous work, such as the English, Hungarian, Icelandic, and Swedish datasets (e.g., Pettersson et al., 2014; Pettersson, 2016; Robertson and Goldwater, 2018; Tang et al., 2018) and the Slovene datasets (e.g., Ljubeic et al., 2016; Scherrer and Erjavec, 2016; Etxeberria et al., 2016; Domingo and Casacuberta, 2018). Additionally, contemporary datasets are required for the rule-based and distance-based components of Norma, as they expect a list of valid target word forms to function properly. For this, we want to choose resources that are readily available for many languages and are reliable, i.e., consist of carefully edited text. Here, I choose a combination of three sources: 7 (i) the normalizations in the training sets, (ii) the Europarl corpus (Koehn, 2005), and (iii) the parallel Bible corpus by Christodouloupoulos and Steedman (2015). The only exception is Icelandic, which is not covered by Europarl; here, we can follow Pettersson (2016) instead by using data from two specialized resources, the BN database (Bjarnadt-tir, 2012) and the MM corpus (Helgadttir et al., 2012). This way, we obtain full-form lexica of 7 Detailed descriptions of the data extraction procedure can be found in the Supplementary Material. 12k64k word types from the Bible corpus, 55k 268k types from Europarl, and 2.8M types from the Icelandic resources. Preprocessing The most important preprocessing decisions 8 are (i) to lowercase all characters and (ii) to remove all punctuation-only tokens. Both capitalization and punctuation often cannot be handled correctly without considering token context, which all current normalization models do not do. Furthermore, their usage can be very erratic in historical texts, potentially distorting the evaluation; e.g., when a text uses punctuation marks according to modern conventions, their normalization is usually trivial, resulting in artificial gains in normalization accuracy that other texts do not get. At the same time, most previous work has not followed these same preprocessing guidelines, making a direct comparison more difficult. This work tries to make up for this by evaluating many different systems, effectively reproducing some of these previous results instead. All models are trained and evaluated separately for each dataset by calculating word accuracy over all tokens. In particular, there is no step to discriminate between tokens that require normalization and those that do not; all word forms in the datasets are treated equally. 8 The full preprocessing steps can be found in the Supplementary Material. component, while the latter is reported to produce the best results (Bollmann, 2012). For cSMTiser, the authors suggest using additional monolingual data to improve the language model; the contemporary datasets are used for this purpose and the model is trained both without and with this additional data; the latter is denoted cSMTiser +LM . For NMT, the model by Bollmann (2018) is evaluated using an ensemble of five models; the model by Tang et al. (2018) is trained on character-level input using the default settings provided by their implementation. 9 To illustrate how challenging the normalization task is on different datasets, we can additionally look at the identity baseline i.e., the percentage of tokens that do not need to be normalizedas well as the maximum accuracy obtainable if each word type was mapped to its most frequently occurring normalization. The latter gives an indication of the extent of ambiguity in the datasets and the disadvantage of not considering token context (cf. Sec. 2.6). Results Table 2 shows the results of this evaluation. The extent of spelling variation varies 9 This is the Att-RNN setting reported in their paper; due to the high computational demands of the model, it was not feasible to run experiments with multiple configurations. greatly between datasets, with less than 15% of tokens requiring normalization (SLG ) to more than 80% (HU). The maximum accuracy is above 97% for most datasets, suggesting that we can obtain high normalization accuracy in principle even without considering token context. For the normalization systems, we observe significantly better word accuracy with SMT than NMT on four of the datasets, and non-significant differences on five others. There is only one dataset (DEA ) where the NMT system by Tang et al. (2018) gets significantly better word accuracy than other systems. This somewhat contradicts the results from Tang et al. (2018), who find NMT to usually outperform the SMT baseline by Pettersson et al. (2014). However, note that the results for the cSMTiser system are often significantly better than reported in previous work: e.g., on Hungarian, cSMTiser obtains 91.7% accuracy, but only 80.1% with the SMT system from Pettersson et al. (2014). Overall, the deep NMT model by Tang et al. (2018) consistently outperforms the shallow one by Bollmann (2018). cSMTiser seems to ben-efit from the added contemporary data for language modelling, though the effect is not significant on any individual dataset. Finally, while Norma does produce competitive results on sev-Method Dataset DEADEREN ES HU IS PT SLBSLGSV Norma, Lookup 0.41 0.31 0.38 0.35 0.43 0.38 0.39 0.44 0.44 0.29 Norma, Rule-based 0.40 0.33 0.43 0.39 0.38 0.40 0.45 0.47 0.46 0.32 Norma, Distance-based 0.42 0.34 0.46 0.44 0.41 0.44 0.50 0.52 0.39 0.38 Norma (Combined) 0.41 0.33 0.45 0.42 0.34 0.42 0.51 0.51 0.42 0.31 cSMTiser 0.37 0.26 0.39 0.41 0.26 0.40 0.50 0.53 0.56 0.24 cSMTiser +LM 0.39 0.27 0.39 0.42 0.27 0.41 0.50 0.53 0.56 0.24 NMT (Bollmann, 2018) 0.38 0.26 0.39 0.43 0.27 0.40 0.48 0.47 0.51 0.23 NMT (Tang et al., 2018) 0.38 0.27 0.38 0.42 0.26 0.41 0.46 0.50 0.56 0.24 (b) Stemming accuracy: percentage of incorrect normalizations with correct word stems (higher is better) Table 3: Evaluations on the subset of incorrect normalizations only; best results for each dataset in bold. eral datasets (particularly in the combined set-ting), it is generally significantly behind the SMT and NMT methods. While word accuracy is easily interpretable, it is also a very crude measure, as it classifies predictions as correct/incorrect without considering the type of error(s) made by the model. Character error rate (CER) has sometimes been suggested as a complement to address this issue, but I believe this is not very insightful: For any normalization system that achieves a reasonably high word accuracy, CER will highly correlate with accuracy simply because CER equals zero for any word that is accurately normalized. 10 At the same time, there is a need for a more fine-grained way to assess the normalization quality. Consider the follow-10 When comparing word accuracy scores in Table 2 with the same configurations evaluated using CER, they correlate with Pearson's r   0 . 96 . ing example from the Hungarian dataset with its predicted normalization from the NMT system by Bollmann (2018): (1) ORIG yduewzewlendewk GOLD dvzlendoek PRED dvzlendok Here, the prediction matches the correct target form almost perfectly, but would be counted as incorrect since it misses an insertion of the letter (cid:104) e (cid:105) towards the end. In this vein, it will be treated the same by the word accuracy measure as a prediction that, e.g., had left the original form unchanged. CERI One alternative is to consider character error rate on the subset of incorrect normalizations only. This way, CER becomes a true complement to word accuracy by assessing the magnitude of error that a normalization model makes when it is not perfectly accurate. The results of this measure, denoted CERI , are shown in Table 3a. The lowest CERI score is often achieved by Norma's lookup module, which leaves historical word forms unchanged if they are not in its lookup wordlist learned during training. This suggests that the incorrect predictions made by other systems are often worse than just leaving the historical spelling unchanged. Stemming Another problem of CER is that all types of errors are treated the same: a one-letter difference in inflection, such as king  kings or came  come , would be treated identically to an error that changes the meaning of the word ( bids  beds ) or results in a non-word ( creature  crya-ture ). I propose an approach that, to the best of my knowledge, has not been used in normalization evaluation before: measure accuracy on word stems, i.e., process both the reference normalization and the prediction with an automatic stemming algorithm and check if both stems match. For this evaluation, I choose the Snowball stemmer (Porter, 2001) as it contains stemming algorithms for many languages (including the ones represented here except for Icelandic and Slovene) and is publicly available. 11 Table 3b shows the accuracy on word stems, again only evaluated on the subset of incorrect normalizations, as this better highlights the differences between settings. This evaluation reveals some notable differences between datasets : For example, while the English and Spanish datasets have very comparable accuracy scores overall (cf. Tab. 2), they show very different characteristics in the stemming evaluation; for English, only up to 9.86% of incorrect predictions show the correct word stem, while for Spanish the number is up to 43.82%. Examining predictions on the dev set, many of the incorrectly predicted cases in Spanish result from mistakes in placement of diacritics, such as sta  est or enve  envi ; the stemming algorithm removes diacritics and can therefore match these instances. Overall, this gives an indication that the errors made on the Spanish dataset are less severe than those on English, despite comparable word accuracy scores and a usually higher CERI for Spanish. This case study shows that stemming can be a useful tool for error analysis in normalization models and reveal characteristics that neither word accuracy nor CER alone can show. Supervised methods for historical text normalization have been evaluated with highly varying amounts of training data: e.g., Domingo and Casacuberta (2018) train a normalizer for 17 th century Spanish on 436k tokens; Etxeberria et al. (2016) use only 8k tokens to train a normalizer for Basque. Even in the evaluation in Sec. 4, training set sizes varied between 24k and 234k tokens, depending on the dataset. Furthermore, many research projects seeking to use automatic normalization techniques cannot afford to produce training data in high quantity. All of this raises the question how different normalization systems perform with varying amounts of training data, and whether reasonable normalization results can be achieved in a low-resource scenario. Methodology All models are retrained on varying subsets of the training data, with sizes ranging from 100 tokens to 50,000 tokens. However, the lower the training set size is, the higher the potential variance when training on it, since random factors such as the covered spelling variants or vocabulary are more likely to impact the results. Therefore, I choose the following approach: For each dataset and training size, up to ten different training splits are extracted, 12 and a separate model is trained on each one. Each model is then evaluated on the respective development dataset, and only the average accuracy across all splits is considered. Results Figure 2 shows two learning curves that are representative for most of the datasets. 13 They reveal that Norma (in the combined setting) performs best in extremely low-resource scenarios, but is overtaken by the SMT approach as more training data becomes available; usually already around 5001000 tokens. The NMT models have a steeper learning curve, needing more training data to become competitive. Extrapolating this trend, it is conceivable that the NMT models would simply need more training data than our current datasets provide in order to consistently outperform the SMT approach. On the other hand, there appears to be no correlation between the size of the training set (cf. Tab. 1) and the relative per-12 The ten training splits consist of chunks of n tokens that are spaced equidistantly across the full training set; for larger n , the number of chunks is reduced so that no splits overlap to more than 50%. 13 Plots for all datasets can be found in the Appendix. formance of NMT vs. SMT (cf. Tab. 2) in the experiments. Since I am not aware of larger datasets for the historical normalization task, this remains an open question for now. A remarkable result is that very small amounts of training data can already be helpful for the normalization task. The English dataset has comparatively little spelling variation to begin with: leaving all words unnormalized already results in an accuracy of 75.5%. Still, with as little as 100 tokens for training, applying the Norma tool raises the accuracy above 83%. For Hungarian, the same amount of training data raises the accuracy from 17.8% (unnormalized) to around 50%. It would be interesting to further compare these results with fully unsupervised methods. Robertson and Goldwater (2018) highlight the importance of evaluating separately on seen vs. unseen tokens, i.e., tokens that have also been in the training set (in-vocabulary) and those that have not (out-of-vocabulary), as well as comparing to a naive memorization baseline. These numbers are presented in Table 4. For unseen tokens (Tab. 4b), the accuracy scores follow generally the same trend as in the full evaluation of Tab. 2; i.e., SMT performs best in most cases. For seen tokens (Tab. 4a), however, Norma's lookup component which implements naive memorizationobtains the highest score on nine datasets. These observations suggest a new normalization strategy: apply the naive lookup on the subset of in-vocabulary tokens and the SMT/NMT models on the subset of out-of-vocabulary tokens only. Table 5 shows the results of this strategy. 14 On nine datasets, it performs better than always using the learned models (as in Tab. 2), and this difference is statistically significant on five of them. These results support the claim from Robertson and Goldwater (2018) that learned models should typically only be applied to unseen tokens. 6 Conclusion This paper presented a large study of historical text normalization. Starting with a systematic survey of the existing literature, four different systems (based on supervised learning) were evaluated and compared on datasets from eight different languages. On the basis of these results, we can extract some practical recommendations for projects seeking to employ normalization techniques: 1. to use the Norma tool when only little training data ( < 500 tokens) is available; 2. to use cSMTiser otherwise, ideally with additional data for language modelling; and 3. to make use of the naive memoriza-tion/lookup technique for in-vocabulary tokens when possible. Furthermore, the qualitative analysis (in Sec. 5.1) should encourage authors evaluating normalization systems to use task-motivated approaches, such as evaluation on word stems, to provide 14 The non-lookup components of Norma are not included in this evaluation since Norma (Combined) effectively implements such a strategy already. deeper insight into the properties of their models and datasets. Detailed information on how to train and apply all of the evaluated techniques is made available online at https://github.com/ coastalcph/histnorm . I would like to thank my PhD supervisor, Stefanie Dipper, for her continuous support over many years that culminated in the doctoral thesis on which this paper is based; the acknowledgments in that thesis largely extend to this paper as well. Many thanks to Anders Sgaard for many helpful discussions and for supporting the follow-up experiments conducted for this paper. Further thanks go to the anonymous reviewers whose helpful suggestions have largely been incorporated here. I gratefully acknowledge the donation of a Titan Xp GPU by the NVIDIA Corporation that was used for a substantial part of this research.
abstain abstain method objective abstain abstain abstain abstain method abstain abstain objective abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other other abstain abstain other abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other