MaskLID: Code-Switching Language Identification through Iterative
  Masking
We present MaskLID, a simple, yet effective, code-switching (CS) language
identification (LID) method. MaskLID does not require any training and is
designed to complement current high-performance sentence-level LIDs.
Sentence-level LIDs are classifiers trained on monolingual texts to provide
single labels, typically using a softmax layer to turn scores into
probabilities. However, in cases where a sentence is composed in both L1 and L2
languages, the LID classifier often only returns the dominant label L1. To
address this limitation, MaskLID employs a strategy to mask text features
associated with L1, allowing the LID to classify the text as L2 in the next
round. This method uses the LID itself to identify the features that require
masking and does not rely on any external resource. In this work, we explore
the use of MaskLID for two open-source LIDs (GlotLID and OpenLID), that are
both based on the FastText architecture. Code and demo are available at
https://github.com/cisnlp/MaskLID.
666840d3bb4e8a77538b1300
Amir Hossein Kargaran
claimed_verified
2024-06-11T13:26:27.595Z
61bf84c8ca59d6d196a1b4e8
https://cdn-avatars.huggingface.co/v1/production/uploads/61bf84c8ca59d6d196a1b4e8/L_NvUwlMYcye9X35z6f7e.jpeg
Amir Hossein Kargaran
user
kargaranamir
666840d3bb4e8a77538b1301
François Yvon
666840d3bb4e8a77538b1302
Hinrich Schütze
MaskLID: Code-Switching Language Identification through Iterative
  Masking
We present MaskLID, a simple, yet effective, code-switching (CS) language
identification (LID) method. MaskLID does not require any training and is
designed to complement current high-performance sentence-level LIDs.
Sentence-level LIDs are classifiers trained on monolingual texts to provide
single labels, typically using a softmax layer to turn scores into
probabilities. However, in cases where a sentence is composed in both L1 and L2
languages, the LID classifier often only returns the dominant label L1. To
address this limitation, MaskLID employs a strategy to mask text features
associated with L1, allowing the LID to classify the text as L2 in the next
round. This method uses the LID itself to identify the features that require
masking and does not rely on any external resource. In this work, we explore
the use of MaskLID for two open-source LIDs (GlotLID and OpenLID), that are
both based on the FastText architecture. Code and demo are available at
https://github.com/cisnlp/MaskLID.
MaskLID, a code-switching language identification method, enhances sentence-level LIDs by masking dominant language features to identify mixed-language sentences without additional training.
code-switching language identification sentence-level LIDs MaskLID fastText GlotLID OpenLID