title: NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for
    Local Communities
summary: Enhancing the linguistic capabilities of Large Language Models (LLMs) to
  include low-resource languages is a critical research area. Current research
  directions predominantly rely on synthetic data generated by translating
  English corpora, which, while demonstrating promising linguistic understanding
  and translation abilities, often results in models aligned with source language
  culture. These models frequently fail to represent the cultural heritage and
  values of local communities. This work proposes a methodology to create both
  synthetic and retrieval-based pre-training data tailored to a specific
  community, considering its (i) language, (ii) cultural heritage, and (iii)
  cultural values. We demonstrate our methodology using Egyptian and Moroccan
  dialects as testbeds, chosen for their linguistic and cultural richness and
  current underrepresentation in LLMs. As a proof-of-concept, we develop
  NileChat, a 3B parameter LLM adapted for Egyptian and Moroccan communities,
  incorporating their language, cultural heritage, and values. Our results on
  various understanding, translation, and cultural and values alignment
  benchmarks show that NileChat outperforms existing Arabic-aware LLMs of similar
  size and performs on par with larger models. We share our methods, data, and
  models with the community to promote the inclusion and coverage of more diverse
  communities in LLM development.
paper_authors._id: 6835043630e39ad8b0fa3f01
paper_authors.name: Abdellah El Mekki
paper_authors._id: 6835043630e39ad8b0fa3f02
paper_authors.name: Houdaifa Atou
paper_authors._id: 6835043630e39ad8b0fa3f03
paper_authors.name: Omer Nacar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:44:30.737Z
paper_authors._id: 628f7a71dd993507cfcbe587
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628f7a71dd993507cfcbe587/3frAcMgfBCx-OJsrdAhnb.png
paper_authors.fullname: Omartificial Intelligence Space
paper_authors.type: user
paper_authors.user: Omartificial-Intelligence-Space
paper_authors._id: 6835043630e39ad8b0fa3f04
paper_authors.name: Shady Shehata
paper_authors._id: 6835043630e39ad8b0fa3f05
paper_authors.name: Muhammad Abdul-Mageed
paper_title: NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for
    Local Communities
paper_summary: Enhancing the linguistic capabilities of Large Language Models (LLMs) to
  include low-resource languages is a critical research area. Current research
  directions predominantly rely on synthetic data generated by translating
  English corpora, which, while demonstrating promising linguistic understanding
  and translation abilities, often results in models aligned with source language
  culture. These models frequently fail to represent the cultural heritage and
  values of local communities. This work proposes a methodology to create both
  synthetic and retrieval-based pre-training data tailored to a specific
  community, considering its (i) language, (ii) cultural heritage, and (iii)
  cultural values. We demonstrate our methodology using Egyptian and Moroccan
  dialects as testbeds, chosen for their linguistic and cultural richness and
  current underrepresentation in LLMs. As a proof-of-concept, we develop
  NileChat, a 3B parameter LLM adapted for Egyptian and Moroccan communities,
  incorporating their language, cultural heritage, and values. Our results on
  various understanding, translation, and cultural and values alignment
  benchmarks show that NileChat outperforms existing Arabic-aware LLMs of similar
  size and performs on par with larger models. We share our methods, data, and
  models with the community to promote the inclusion and coverage of more diverse
  communities in LLM development.
paper_ai_summary: A methodology is proposed to create pre-training data tailored to low-resource languages and cultures, demonstrated through NileChat, a 3B parameter LLM for Egyptian and Moroccan dialects, which outperforms existing similar-sized Arabic-aware models.
paper_ai_keywords: Large Language Models synthetic data retrieval-based pre-training cultural heritage cultural values NileChat