title: Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of
    Small Multilingual Language Models for Low-Resource Languages
summary: Low-resource languages (LRLs) face significant challenges in natural language
  processing (NLP) due to limited data. While current state-of-the-art large
  language models (LLMs) still struggle with LRLs, smaller multilingual models
  (mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of
  their capacity to low training data sizes. This study systematically
  investigates parameter-efficient adapter-based methods for adapting mLMs to
  LRLs, evaluating three architectures: Sequential Bottleneck, Invertible
  Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and
  structured knowledge from ConceptNet, we show that small adaptation datasets
  (e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains
  in intrinsic (masked language modeling) and extrinsic tasks (topic
  classification, sentiment analysis, and named entity recognition). We find that
  Sequential Bottleneck adapters excel in language modeling, while Invertible
  Bottleneck adapters slightly outperform other methods on downstream tasks due
  to better embedding alignment and larger parameter counts. Adapter-based
  methods match or outperform full fine-tuning while using far fewer parameters,
  and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3,
  GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves
  performance, pre-training data size remains the dominant factor, especially for
  languages with extensive pre-training coverage.
paper_authors._id: 67b32e9dff65b4ec02cb6d81
paper_authors.name: Daniil Gurgurov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T16:39:55.120Z
paper_authors._id: 6427f45beb320ead3d287acf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6427f45beb320ead3d287acf/J7DLdQM6ehiJBke112s0q.jpeg
paper_authors.fullname: Daniil Gurgurov
paper_authors.type: user
paper_authors.user: DGurgurov
paper_authors._id: 67b32e9dff65b4ec02cb6d82
paper_authors.name: Ivan Vykopal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-17T12:42:07.190Z
paper_authors._id: 64b7e5a89e7deb6a7824301b
paper_authors.avatarUrl: /avatars/34d8843fe6203c4d09bf1b442405a4ab.svg
paper_authors.fullname: Ivan Vykopal
paper_authors.type: user
paper_authors.user: ivykopal
paper_authors._id: 67b32e9dff65b4ec02cb6d83
paper_authors.name: Josef van Genabith
paper_authors._id: 67b32e9dff65b4ec02cb6d84
paper_authors.name: Simon Ostermann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T10:00:05.016Z
paper_authors._id: 664de652b3eaebda418c4ce9
paper_authors.avatarUrl: /avatars/7f35afa838566daaeebb6e2915013871.svg
paper_authors.fullname: Simon Ostermann
paper_authors.type: user
paper_authors.user: simonost
paper_title: Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of
    Small Multilingual Language Models for Low-Resource Languages
paper_summary: Low-resource languages (LRLs) face significant challenges in natural language
  processing (NLP) due to limited data. While current state-of-the-art large
  language models (LLMs) still struggle with LRLs, smaller multilingual models
  (mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of
  their capacity to low training data sizes. This study systematically
  investigates parameter-efficient adapter-based methods for adapting mLMs to
  LRLs, evaluating three architectures: Sequential Bottleneck, Invertible
  Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and
  structured knowledge from ConceptNet, we show that small adaptation datasets
  (e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains
  in intrinsic (masked language modeling) and extrinsic tasks (topic
  classification, sentiment analysis, and named entity recognition). We find that
  Sequential Bottleneck adapters excel in language modeling, while Invertible
  Bottleneck adapters slightly outperform other methods on downstream tasks due
  to better embedding alignment and larger parameter counts. Adapter-based
  methods match or outperform full fine-tuning while using far fewer parameters,
  and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3,
  GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves
  performance, pre-training data size remains the dominant factor, especially for
  languages with extensive pre-training coverage.
paper_ai_keywords: adapter-based methods Sequential Bottleneck Invertible Bottleneck Low-Rank Adaptation masked language modeling topic classification sentiment analysis named entity recognition Adaptation Datasets GlotCC ConceptNet embedding alignment full fine-tuning mBERT XLM-R LLaMA-3 GPT-4 DeepSeek-R1-based distilled models