This work presents a new resource for borrowing identification and analyzes the performance and errors of several models on this task. We introduce a new annotated corpus of Spanish newswire rich in unassimilated lexical borrowingswords from one language that are introduced into another without orthographic adaptationand use it to evaluate how several sequence labeling models (CRF, BiLSTM-CRF, and Transformer-based models) perform. The corpus contains 370,000 tokens and is larger, more borrowing-dense, OOV-rich, and topic-varied than previous corpora available for this task. Our results show that a BiLSTM-CRF model fed with subword embeddings along with either Transformer-based embeddings pretrained on codeswitched data or a combination of contextualized word embeddings outperforms results obtained by a multilingual BERT-based model. Lexical borrowing is the process of bringing words from one language into another (Haugen, 1950). Borrowings are a common source of out-of-vocabulary (OOV) words, and the task of detecting borrowings has proven to be useful both for lexicographic purposes and for NLP downstream tasks such as parsing (Alex, 2008a), text-to-speech synthesis (Leidig et al., 2014), and machine translation (Tsvetkov and Dyer, 2016). Recent work has approached the problem of extracting lexical borrowings in European languages such as German (Alex, 2008b; Garley and Hocken-maier, 2012; Leidig et al., 2014), Italian (Furiassi and Hofland, 2007), French (Alex, 2008a; Ches-ley, 2010), Finnish (Mansikkaniemi and Kurimo, 2012), Norwegian (Andersen, 2012; Losnegaard and Lyse, 2012), and Spanish (Serigos, 2017), with a particular focus on English lexical borrowings (often called anglicisms ). Computational approaches to mixed-language data have traditionally framed the task of identifying the language of a word as a tagging problem, where every word in the sequence receives a language tag (Lignos and Marcus, 2013; Molina et al., 2016; Solorio et al., 2014). As lexical borrowings can be single (e.g. app , online , smartphone ) or multi-token (e.g. machine learning ), they are a natural fit for chunking-style approaches. lvarez Mellado (2020b) introduced chunking-based models for borrowing detection in Spanish media which were later improved (lvarez Mellado, 2020a), producing an F1 score of 86.41. However, both the dataset and modeling approach used by lvarez Mellado (2020a) had significant limitations. The dataset focused exclusively on a single source of news and consisted only of headlines. The number and variety of borrowings were limited, and there was a significant overlap in borrowings between the training set and the test set, which prevented assessment of whether the modeling approach was actually capable of generalizing to previously unseen borrowings. Additionally, the best results were obtained by a CRF model, and more sophisticated approaches were not explored. The contributions of this paper are a new corpus of Spanish annotated with unassimilated lexical borrowings and a detailed analysis of the performance of several sequence-labeling models trained on this corpus. The models include a CRF, Transformer-based models, and a BiLSTM-CRF with different word and subword embeddings (in-cluding contextualized embeddings, BPE embeddings, character embeddings, and embeddings pretrained on codeswitched data). The corpus contains 370,000 tokens and is larger and more topic-varied than previous resources. The test set was designed to be as difficult as possible; it covers sources and dates not seen in the training set, includes a high number of OOV words (92% of the borrowings in the test set are OOV) and is very borrowing-dense 3868 Media Topics Set(s) ElDiario.es General newspaper Train, Dev. The dataset we present is publicly available 1 and has been released under a CC BY-NC-SA-4.0 license. This dataset was used for the ADoBo shared task on automatic detection of borrowings at IberLEF 2021 (lvarez Mellado et al., 2021). Linguistic borrowing can be defined as the transference of linguistic elements between two languages. Borrowing and codeswitching have been described as a continuum (Clyne et al., 2003). Lexical borrowing involves the incorporation of single lexical units from one language into another language and is usually accompanied by morphological and phonological modification to conform with the patterns of the recipient language (Onysko, 2007; Poplack et al., 1988). On the other hand, codeswitches are by definition not integrated into a recipient language, unlike established loanwords (Poplack, 2012). While codeswitches require a substantial level of fluency, comply with grammatical restrictions in both languages, and are produced by bilingual speakers in bilingual discourses, lexical borrowings are words used by monolingual individuals that eventually 1 https://github.com/lirondos/coalas Set Tokens ENG OTHER Unique Training 231,126 1,493 28 380 Development 82,578 306 49 316 Test 58,997 1,239 46 987 Total 372,701 3,038 123 1,683 Table 2: Corpus splits with counts become lexicalized and assimilated as part of the recipient language lexicon until the knowledge of foreign disappears (Lipski, 2005). Our dataset consists of Spanish newswire annotated for unassimilated lexical borrowings. All of the sources used are European Spanish online publications (newspapers, blogs, and news sites) published in Spain and written in European Spanish. Data was collected separately for the training, development, and test sets to ensure minimal overlap in borrowings, topics, and time periods. The training set consists of a collection of articles appearing between August and December 2020 in elDiario.es , a progressive online newspaper based in Spain. The development set contains sentences in articles from January 2021 from the same source. The data in the test set consisted of annotated sentences extracted in February and March 2021 from a diverse collection of online Spanish media that covers specialized topics rich in lexical borrowings and usually not covered by elDiario.es , such as sports, gossip or videogames (see Table 1). To focus annotation efforts for the training set on articles likely to contain unassimilated borrowings, the articles to be annotated were selected by first using a baseline model and were then human-annotated. To detect potential borrowings, the CRF model and data from lvarez Mellado (2020b) was used along with a dictionary look-up pipeline. Articles that contained more than 5 borrowing candidates were selected for annotation. The main goal of data selection for the development and test sets was to create borrowing-dense, OOV-rich datasets, allowing for better assessment of generalization. To that end, the annotation was based on sentences instead of full articles. If a sentence contained a word either flagged as a borrowing by the CRF model, contained in a wordlist of English, or simply not present in the training set, it was selected for annotation. This data selection approach ensured a high number of borrowings and 3869 OOV words, both borrowings and non-borrowings. While the training set contains 6 borrowings per 1,000 tokens, the test set contains 20 borrowings per 1,000 tokens. Additionally, 90% of the unique borrowings in the development set were OOV (not present in training). 92% of the borrowings in the test set did not appear in training (see Table 2). The corpus was annotated with BIO encoding using Doccano (Nakayama et al., 2018) by a native speaker of Spanish with a background in linguistic annotation (see Appendix C). The annotation guidelines (provided in Appendix B) were based on those of lvarez Mellado (2020a) but were expanded to account for a wider diversity of topics. Following Serigos's observations and lvarez Mel-lado's work, English lexical borrowings were labeled ENG , other borrowings were labeled OTHER . Here is an example from the training set: 2 Benching [ ENG ], estar en el banquillo de tu crush [ ENG ] mientras otro juega de titular. In order to assess the quality of the guidelines and the annotation, a sample of 9,110 tokens from 450 sentences (60% from the test set, 20% from training, 20% from development) was divided among a group of 9 linguists for double annotation. The mean inter-annotation agreement computed by Cohen's kappa was 0.91, which is above the 0.8 threshold of reliable annotation (Artstein and Poesio, 2008). Like all resources, this resource has significant limitations that its users should be aware of. The corpus consists exclusively of news published in Spain and written in European Spanish. This fact by no means implies the assumption that European Spanish represents the whole of the Spanish language. The notion of assimilation is usage-based and community-dependant, and thus the dataset we present and the annotation guidelines that were followed were designed to capture a very specific phenomena at a given time and in a given place: unassimilated borrowings in the Spanish press. In order to establish whether a given word has been assimilated or not, the annotation guidelines rely on lexicographic sources such as the prescriptivist Diccionario de la Lengua Espaola (Real Academia Espaola, 2020) by the Royal 2 Benching: being on your crush's bench while someone else plays in the starting lineup. Spanish Academy, a dictionary that aims to cover world-wide Spanish but whose Spain-centric criteria has been previously pointed out (Blanch, 1995; Fernndez Gordillo, 2014). In addition, prior work has suggested that Spanish from Spain may have a higher tendency of anglicism-usage than other Spanish dialects (McClelland, 2021). Consequently, we limit the scope of the dataset to European Spanish not because we consider that this variety represents the whole of the Spanish-speaking community, but because we consider that the approach we have taken here may not account adequately for the whole diversity in borrowing assimilation within the Spanish-speaking world. Our annotation is licensed with a permissive CC BY-NC-SA-4.0 license. With one exception, the sources included in our dataset release their content under Creative Commons licenses that allow for reusing and redistributing the material for non commercial purposes. This was a major point when deciding which news sites would be included in the dataset. The exception is the source Microsiervos , whose content we use with explicit permission from the copyright holder. Our annotation is stand-off annotation that does not create a derivative work under Creative Commons licenses, so ND licenses are not a problem for our resource. Table 9 in Appendix A lists the URL and license type for each source. The corpus was used to evaluate four types of models for borrowing extraction: (1) a CRF model, (2) two Transformer-based models, (3) a BiLSTM-CRF model with different types of unadapted embeddings (word, BPE, and character embeddings) and (4) a BiLSTM-CRF model with previously fine-3870 tuned Transformer-based embeddings pretrained on codeswitched data. By unadapted embeddings, we mean embeddings that have not been fine-tuned for the task of anglicism detection or a related task (e.g. codeswitching). Evaluation for all models required extracted spans to match the annotation exactly in span and type to be correct. Evaluation was performed with SeqScore (Palen-Michel et al., 2021), using conlleval -style repair for invalid label sequences. All models were trained using an AMD 2990WX CPU and a single RTX 2080 Ti GPU. As baseline model, we evaluated a CRF model with handcrafted features from lvarez Mellado (2020b). The model was built using pycrfsuite (Peng and Korobov, 2014), a Python wrapper for crfsuite (Okazaki, 2007) that implements CRF for labeling sequential data. The model also uses the Token and Span utilities from spaCy library (Honnibal and Montani, 2017). The following handcrafted binary features from lvarez Mellado (2020b) were used for the model:  Bias: active on all tokens to set per-class bias  Token: the string of the token  Uppercase: active if the token is all uppercase  Titlecase: active if only the first character of the token is capitalized  Character trigram: an active feature for every trigram contained in the token  Quotation: active if the token is any type of quotation mark ( ' "    )  Suffix: last three characters of the token  POS tag: part-of-speech tag of the token provided by spaCy utilities  Word shape: shape representation of the token provided by spaCy utilities  Word embedding: provided by Spanish word2vec 300 dimensional embeddings by Cardellino (2019), one feature per dimension  URL: active if the token could be validated as a URL according to spaCy utilities  Email: active if the token could be validated as an email address by spaCy utilities  Twitter: active if the token could be validated as a possible Twitter special token: #hashtag or @username A window of two tokens in each direction was used for feature extraction. Optimization was performed using L-BFGS, with the following hyper-parameter values chosen following the best results from lvarez Mellado (2020b) were set: c1 = 0 . 05 , c2 = 0 . 01 . As shown in Table 3, the CRF produced an overall F1 score of 66.15 on the development set (P: 74.13, R: 59.72) and an overall F1 of 55.44 (P: 77.89, R: 43.04) on the test set. The CRF results on our dataset are far below the F1 of 86.41 reported by lvarez Mellado (2020b), showing the impact that a topically-diverse, OOV-rich dataset can have, especially on test set recall. These results demonstrate that we have created a more difficult task and motivate using more sophisticated models. model trained for Spanish (Caete et al., 2020)  mBERT: multilingual BERT, trained on Wikipedia in 104 languages (Devlin et al., 2019) Both models were run using the Transformers library by HuggingFace (Wolf et al., 2020). The same default hyperparameters were used for both models: 3 epochs, batch size 16, and maximum sequence length 256. Except where otherwise specified, we report results for 10 runs that use different random seeds for initialization. We perform statistical significance testing using the Wilcoxon rank-sum test. (F1: 82.02) performed better than BETO (F1: 80.71), and the difference was statistically significant ( p = 0 . 027 ). Both models performed better on the test set than on the development set, despite the difference in topics between them, suggesting good generalization. This is a remarkable difference with the CRF results, where the CRF performed substantially worse on the test set than on the development set. The limited number of OTHER examples explains the high deviations in the results for this label. We explored several possibilities for a BiLSTM-CRF model fed with different types of word and subword embeddings. The purpose was to assess whether the combination of different embeddings that encode different linguistic information could outperform the Transformer-based models in Section 3.2. All of our BiLSTM-CRF models were built using Flair (Akbik et al., 2018) with default hyperparameters (hidden size = 256, learning rate = 0.1, mini batch size = 32, max number of epochs = 150) and embeddings provided by Flair . We first ran exploratory experiments on the development set with different types of embeddings using Flair tuning functionalities. We explored the following embeddings: Transformer embeddings (mBERT and BETO), fastText embeddings (Bo-janowski et al., 2017), one-hot embeddings, byte pair embeddings (Heinzerling and Strube, 2018), and character embeddings (Lample et al., 2016). The best results were obtained by a combination of mBERT embeddings and character embeddings (F1: 74.00), followed by a combination of BETO embeddings and character embeddings (F1: 72.09). These results show that using contextualized embeddings unsurprisingly outperforms non-contextualized embeddings for this task, and that subword representation is important for the task of extracting borrowings that have not been adapted orthographically. The finding regarding the importance of subwords is consistent with previous work; feature ablation experiments for borrowing detection have shown that character trigram features contributed the most to the results obtained by a CRF model (lvarez Mellado, 2020b). The worst result (F1: 39.21) was produced by a model fed with one-hot vectors, and the second-worst result was produced by a model fed exclusively with character embeddings. While it performed poorly (F1: 41.65), this fully unlexicalized model outperformed one-hot embeddings, reinforcing the importance of subword information for the task of unassimilated borrowing extraction. In light of the preliminary embedding experiments and our earlier experiments with Transformer-based models, we fed our BiLSTM-CRF model with different combinations of contextualized word embeddings (including English BERT embeddings from Devlin et al.), byte-pair embeddings and character embeddings. Table 5 shows development set results from different combinations of embeddings. The best overall F1 on the development set was obtained by the combination of BETO embeddings, BERT embeddings and byte-pair embeddings. The model fed with BETO embeddings, BERT embeddings, byte-pair embeddings and character embeddings ranked second. Several things stand out from the results in Table 5. The BETO+BERT embedding combination consistently works better than mBERT embeddings, and BPE embeddings contribute to better results. Character embeddings, however, seem to produce little effect at first glance. Given the same model, adding character embeddings produced little changes in F1 or even slightly hurt the results. Although character embeddings seem to make little difference in overall F1, recall was consistently higher in models that included character embeddings, and in fact, the model with BETO+BERT embeddings, BPE embeddings and character embeddings produced the highest recall overall (77.46). This is an interesting finding, as our results from Sections 3.1 and 3.2 as well as prior work (lvarez Mellado, 2020b) identified recall as weak for borrowing detection models. The two best-performing models from Table 5 (BETO+BERT embeddings, BPE embeddings and optionally character embeddings) were evaluated on the test set. Table 6 gives results per type on the development and test sets for these two models. For both models, results on the test set were better (F1: 82.92, F1: 83.63) than on the development set (F1: 81.21, F1: 81.05). Although the best F1 score on the development set was obtained with no character embeddings, when run on the test set the model with character embeddings obtained the best score; however, these differences did not show to be statistically significant. Recall, on the other hand, 3872 Word embedding BPE embedding Char embedding Precision Recall F1 mBERT -82 . was higher when run with character embeddings (R: 78.34) than when run without them (R: 76.89), and the difference was statistically significant ( p = 0 . 019 ). This finding again corroborates the positive impact that character information can have in recall when dealing with previously unseen borrowings. Finally, we decided to explore whether detecting unassimilated lexical borrowings could be framed as transfer learning from language identification in codeswitching. As before, we ran a BiLSTM-CRF model using Flair , but instead of using the unadapted Transformer embeddings, we used codeswitch embeddings (Sarker, 2020), fine-tuned Transformer-based embeddings pretrained for language identification on the Spanish-English section of the LinCE dataset (Aguilar et al., 2020). Table 7 gives results for these models. The two best-performing models were the BiLSTM-CRF with codeswitch and BPE embeddings (F1: 84.06) and the BiLSTM-CRF model with codeswitch, BPE and character embeddings (F1: 84.22). The differences between these two models did not show to be statistically significant, but the difference with the best-performing model with unadapted embeddings from Section 3.3 (F1: 83.63) was statistically significant ( p = 0 . 018 ). These two best-performing models however obtained worse results on the development set than those obtained by the best-performing models from Section 3.3. Adding BPE embeddings showed to improve F1 score by around 1 point compared to either feeding the model with only codeswitch (F1: 82.83) or only codeswitch and character embeddings (F1: 83.13), and the differences were statistically significant in both cases ( p = 0 . 024 , p = 0 . 018 ). It should be noted that this transfer learning approach is indirectly using more data than just the training data from our initial corpus, as the codeswitch-based BiLSTM-CRF models benefit from the labeled data seen during pretraining for the language-identification task. We compared the different results produced by the best performing model of each type on the test set: (1) the mBERT model, (2) the BiLSTM-CRF with BERT+BETO, BPE and character embeddings and (3) the BiLSTM-CRF model with codeswitch, BPE and character embeddings. We divide the error analysis into two sections. We first analyze errors that were made by all three models, with the aim of discovering which instances of the dataset were 3873 Embeddings Development Test Precision Recall F1 Precision Recall F1 Codeswitch ALL 80 . challenging for all models. We then analyze unique answers (both correct and incorrect) per model, with the aim of gaining insight on what are the unique characteristics of each model in comparison with other models. There were 137 tokens in the test set that were incorrectly labeled as O by all three models. 103 of these were of type ENG , 34 were of type OTHER . These errors can be classified as follows  Borrowings in upper case (12), which tend to be mistaken by models with proper nouns: Anlisis de empresa basados en Big Data [ ENG ]. 3  Borrowings in sentence-initial position (9), which were titlecased and therefore consistently mislabeled as O : Youtuber [ ENG ], mujer y afroamericana: Candace Owen podra ser la alternativa a Trump. 4 Sentence-initial borrowings are particularly tricky, as models tend to confuse these with foreign named entities. In fact, prior work on anglicism detection based on dictionary lookup (Serigos, 2017) stated that borrowings in sentence-initial position were rare in Spanish and consequently chose to ignore all foreign words in sentence-initial position under the assumption that they could be considered named entities. However, these examples (and the 3 Business analytics based on Big Data 4 Youtuber, woman and African-American: Candace Owen could be the alternative to Trump difficulty they pose for models) prove that sentence-initial borrowings are not rare and therefore should not be overlooked. Borrowings that also happen to be words in Spanish (8), such as the word primer , that is a borrowing found in makeup articles ( un primer hidratante , a hydrating primer) but also happens to be a fully Spanish adjective meaning first ( primer premio , first prize). Borrowings like these are still treated as fully unassimilated borrowings by speakers, even when the form is exactly the same as an already-existing Spanish word and were a common source of mislabeling, especially partial mismatches in multitoken borrowings: red (which exists in Spanish meaning net) in red carpet , tractor in tractor pulling or total in total look . Borrowings that could pass as Spanish words (58): most of the misslabeled borrowings were words that do not exist in Spanish but that could orthographically pass for a Spanish word. That is the case of words like burpees (hypothetically, a conjugated form of the non-existing verb burpear ), gimbal , mules , bromance or nude . Other borrowings (50): a high number of mislabeled borrowings were borrowings that were orthographically implausible in Spanish, such as trenchs , multipads , hypes , riff , scrunchie or mint . The fact that none of our models were able to correctly classify these orthographically implausible examples leaves the door open to further exploration of character-based models and investigating character-level perplexity as a source of information. 4.1.2 Non-borrowings labeled as borrowings 29 tokens were incorrectly labeled as borrowings by all three models. These errors can be classified in the following groups:  Metalinguistic usage and reported speech: a foreign word or sentence that appears in the text to refer to something someone said or wrote. Escribir  icon pack  [ ENG ] en el buscador. 5  Lower-cased proper nouns: such as websites. Acceder a la pgina flywithkarolg [ ENG ] 6  Computer commands: the test set included blog posts about technology, which mentioned computer commands (such as sudo apt-get update ) that were consistently mistaken by our models as borrowings. La serie 10.000 ships [ ENG ] cuenta la odisea de la princesa Nymeria. 7  Acronyms and acronym expansions: El entrenamiento HITT ( high intensity interval training [ ENG ]) 8  Assimilated borrowings: certain borrowings that are already considered by RAE's dictionary as fully assimilated were labeled by all models as anglicisms. These may seem like an extreme caseafter all, computer commands do contain English words but they are a good example of the real data that a borrowing-detection system may encounter. Foreign words within proper nouns: lower-cased foreign words that were part of multitoken proper nouns. Three tokens of type OTHER were marked by all models as ENG . There were no ENG borrowings that were labeled as OTHER by all three models. Haba buffet [ ENG ] libre. 10 4.2 Unique answers per model We now summarize the unique mistakes and correct answers made per model, with the aim of understanding what data points were handled uniquely well or badly by each model. There were 46 tokens that were incorrectly labeled as borrowings only by the mBERT model. These include foreign words used in reported speech or acronym expansion (21), proper names (11) and already assimilated borrowings (7). There were 27 tokens that were correctly labeled only by the mBERT model. The mBERT model was particularly good at detecting the full span of multitoken borrowings as in no knead bread , total white , wide leg or kettlebell swings (which were only partially detected by other models) and at detecting borrowings that could pass for Spanish words (such as fashionista , samples , vocoder ). In addition, the mBERT model also correctly labeled as O 12 tokens that the other two models mistook as borrowings, including morphologically adapted anglicisms, such as craftear (Spanish infinitive of the verb to craft ), crackear (from to crack ) or lookazo (augmentative of the noun look ). There were 23 tokens that were incorrectly labeled as borrowings solely by this model, the most common types being assimilated borrowings (such as fan , clon ) and Spanish words ( fiestones ) (9 each). 32 tokens were correctly labeled as borrowings only by this model. These include borrowings that could pass for Spanish words ( camel , canvas ). In addition, this model also correctly labeled as O 6 tokens that the other two mistook as borrowings, including old borrowings that are considered today as fully assimilated (such as films or sake ) or the usage of post as a prefix of Latin origin (as in post-produccin ), which other models mistook with the English word post . The codeswitch-based system incorrectly labeled 18 tokens as borrowings, including proper nouns (7), such as Baby Spice , and fully asimilated borrowings (5), such as jersey , relax or tutorial . This model correctly labeled 27 tokens that were mistakenly ignored by other models, including multitoken borrowings ( dark and gritty , red carpet ) and other borrowings that were non-compliant with Spanish orthographic rules but that were however ignored by other models ( messy , athleisure , multi-touch , workaholic ). The codeswitch-based model also correctly labeled as O 16 tokens that the other two models labeled as borrowings, including acronym expansions, lower-cased proper names and orthographically unorthodox Spanish words, such as the ideophone tiki-taka or shavales (a non-standard writing form of the word chavales , guys). Table 8 provides a summary of our results. As we have seen, the diversity of topics and the presence of OOV words in the dataset can have a remarkable impact on results. The CRF modelwhich in previous work had reported an F1 score of 86saw its performance drop to a 55 when dealing with our dataset, despite the fact that both datasets consisted of journalistic European Spanish texts. On the other hand, neural models (Transformer-based and BiLSTM-CRF) performed better. All of them performed better on the test set than on the development set, which shows good generalization ability. The BiLSTM-CRF model fed with different combinations of Transformer-based word embeddings and subword embeddings outperformed multilingual BERT and Spanish monolingual BETO. The model fed with codeswitch, BPE, and character embeddings ranked first and was significantly better than the result obtained by the model fed with BETO+BERT, BPE, and character embeddings. Our error analysis shows that recall was a weak point for all models we examined. Among false negatives, upper-case borrowings (such as Big Data ) and borrowings in sentence-initial position (in titlecase) were frequent, as they tend to be mistaken with named entities. This finding suggests that borrowings with capitalized initial should not be overlooked. Similarly, words that exist both in English and Spanish (like primer or red ) are not rare and were also a common source of error. Adding character embeddings produced a statistically significant improvement in recall, which opens the door to future work. Concurrently with the work presented on this paper, De la Rosa (2021) explored using supplementary training on intermediate labeled-data tasks (such as POS, NER, codeswitching and language identification) along with multilingual Transformer-based models to the task of detecting borrowings. Alternatively, Jiang et al. (2021) used data augmentation to train a CRF model for the same task. We have introduced a new corpus of Spanish newswire annotated with unassimilated lexical borrowings. The test set has a high number of OOV borrowings92% of unique borrowings in the test set were not seen during trainingand is more borrowing-dense and varied than resources previously available. We have used the dataset to explore several sequence labeling models (CRF, BiLSTM-CRF, and Transformer-based models) for the task of extracting lexical borrowings in a high-OOV setting. Results show that a BiLSTM-CRF model fed with Transformer-based embeddings pretrained on codeswitched data along subword embeddings produced the best results (F1: 84.22, 84.06), followed by a combination of contextualized word embeddings and subword embeddings (F1: 83.63). These models outperformed prior models for this task (CRF F1: 55.44) and multilingual Transformer-based models (mBERT F1: 82.02). In this paper we have introduced an annotated dataset and models for detecting unassimilated borrowings in Spanish. The dataset is openly-licensed, and detailed annotation guidelines are provided (Appendix B). Appendix C includes a data statement that provides information regarding the curation rationale, annotator demographics, text characteristics, etc. of the dataset we have presented. We hope these resources will contribute to bringing more attention to borrowing extraction, a task that has been little explored in the field of NLP but that can be of great help to lexicographers and linguists studying language change. However, the resources we have presented should not be considered a full depiction of either the process of borrowing or the Spanish language in general. We have identified four important considerations that any future systems that build off this research should be aware of. The process of borrowing. Borrowing is a complex phenomenon that can manifest at all linguistic levels (phonological, morphological, lexical, syntactic, semantic, pragmatic). This work is exclusively concerned with lexical borrowings. Furthermore, in this work we have taken a synchronic approach to borrowing: we deal with borrowings that are considered as such in a given dialect and at a given point in time. The process of borrowing assimilation is a diachronic process, and the notion of what is perceived as unassimilated can vary across time and varieties. As a result, our dataset and models may not be suitable to account for partially assimilated borrowings or even for unassimilated borrowings in a different time period. Language variety. The dataset we have presented is exclusively composed of European Spanish journalistic texts. In addition, the guidelines we have described were designed to capture a very specific phenomena: unassimilated borrowings in the Spanish press. In fact, the annotation guidelines rely on sources such as Diccionario de la Lengua Espaola , a lexicographic source whose Spain-centric criteria has been previously pointed out (Blanch, 1995; Fernndez Gordillo, 2014). Consequently, the scope of our work is restricted to unassimilated borrowings in journalistic European Spanish. Our dataset and models may not translate adequately to other Spanish-speaking areas or genres. The preeminence of written language. In our work, the notion of what a borrowing is is heavily influenced by how a word is written. According to our guidelines, a word like meeting will be considered unassimilated, while the Spanish form mitin will be considered assimilated. These preferences in writing may indirectly reveal how well-established a loanword is or how foreign it is perceived by the speaker. But it is questionable that these two forms necessarily represent a difference in pronunciation or linguistic status in the speaker's mental lexicon. How a word is written can be helpful for the purpose of detecting novel anglicisms in written text, but ideally one would not establish a definition of borrowing solely based on lexicographic, corpus-derived or orthotypographic cues. These are all valuable pieces of information, but they only represent an indirect evidence of the status that the word holds in the lexicon. After all, speakers will identify a word as an anglicism (and use it as such), regardless of whether the word is written in a text or used in speech. On the other hand, the lexicographic fact that a word came from another language may not be enough as a criterion to establish the notion of borrowing. Speakers use words all the time without necessarily knowing where they came from or how long ago they were incorporated into the language. The origin of the word may just be a piece of trivia that is totally irrelevant or unknown to the speaker at the time of speaking, so the etymological origin of the word might not be enough to account for the difference among borrowings. In fact, what lies at the core of the unassimilated versus assimilated distinction is the awareness of speakers when they use a certain word (Poplack et al., 1988). The notion of what a borrowing is lies within the brain of the speaker, and in this work we are only indirectly observing that status through written form. Therefore our definition of borrowing and assimilation cannot be regarded as perfect or universal. Ideas about linguistic purity. The purpose of this project is to analyze the usage of borrowings in the Spanish press. This project does not seek to promote or stigmatise the usage of borrowings, or those who use them. The motivation behind our research is not to defend an alleged linguistic purity, but to study the phenomenon of lexical borrowing from a descriptive and data-driven point of view. The authors would like to thank Carlota de Benito Moreno, Jorge Diz Pico, Nacho Esteban Fernndez, Gloria Gil, Clara Manrique, Roco Morn Gonzlez, Aarn Prez Bernabeu, Monserrat Rius and Miguel Snchez Ibez for their assessment of the annotation quality.
objective objective abstain result other other other other other other other other other other objective other other other method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain objective abstain objective abstain abstain abstain abstain method abstain method method abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain other