title: MaskLID: Code-Switching Language Identification through Iterative
    Masking
summary: We present MaskLID, a simple, yet effective, code-switching (CS) language
  identification (LID) method. MaskLID does not require any training and is
  designed to complement current high-performance sentence-level LIDs.
  Sentence-level LIDs are classifiers trained on monolingual texts to provide
  single labels, typically using a softmax layer to turn scores into
  probabilities. However, in cases where a sentence is composed in both L1 and L2
  languages, the LID classifier often only returns the dominant label L1. To
  address this limitation, MaskLID employs a strategy to mask text features
  associated with L1, allowing the LID to classify the text as L2 in the next
  round. This method uses the LID itself to identify the features that require
  masking and does not rely on any external resource. In this work, we explore
  the use of MaskLID for two open-source LIDs (GlotLID and OpenLID), that are
  both based on the FastText architecture. Code and demo are available at
  https://github.com/cisnlp/MaskLID.
paper_authors._id: 666840d3bb4e8a77538b1300
paper_authors.name: Amir Hossein Kargaran
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T13:26:27.595Z
paper_authors._id: 61bf84c8ca59d6d196a1b4e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61bf84c8ca59d6d196a1b4e8/L_NvUwlMYcye9X35z6f7e.jpeg
paper_authors.fullname: Amir Hossein Kargaran
paper_authors.type: user
paper_authors.user: kargaranamir
paper_authors._id: 666840d3bb4e8a77538b1301
paper_authors.name: François Yvon
paper_authors._id: 666840d3bb4e8a77538b1302
paper_authors.name: Hinrich Schütze
paper_title: MaskLID: Code-Switching Language Identification through Iterative
    Masking
paper_summary: We present MaskLID, a simple, yet effective, code-switching (CS) language
  identification (LID) method. MaskLID does not require any training and is
  designed to complement current high-performance sentence-level LIDs.
  Sentence-level LIDs are classifiers trained on monolingual texts to provide
  single labels, typically using a softmax layer to turn scores into
  probabilities. However, in cases where a sentence is composed in both L1 and L2
  languages, the LID classifier often only returns the dominant label L1. To
  address this limitation, MaskLID employs a strategy to mask text features
  associated with L1, allowing the LID to classify the text as L2 in the next
  round. This method uses the LID itself to identify the features that require
  masking and does not rely on any external resource. In this work, we explore
  the use of MaskLID for two open-source LIDs (GlotLID and OpenLID), that are
  both based on the FastText architecture. Code and demo are available at
  https://github.com/cisnlp/MaskLID.
paper_ai_keywords: code-switching (CS) language identification (LID) sentence-level LIDs softmax layer monolingual texts MaskLID text features L1 L2 FastText architecture