NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for
  Local Communities
Enhancing the linguistic capabilities of Large Language Models (LLMs) to
include low-resource languages is a critical research area. Current research
directions predominantly rely on synthetic data generated by translating
English corpora, which, while demonstrating promising linguistic understanding
and translation abilities, often results in models aligned with source language
culture. These models frequently fail to represent the cultural heritage and
values of local communities. This work proposes a methodology to create both
synthetic and retrieval-based pre-training data tailored to a specific
community, considering its (i) language, (ii) cultural heritage, and (iii)
cultural values. We demonstrate our methodology using Egyptian and Moroccan
dialects as testbeds, chosen for their linguistic and cultural richness and
current underrepresentation in LLMs. As a proof-of-concept, we develop
NileChat, a 3B parameter LLM adapted for Egyptian and Moroccan communities,
incorporating their language, cultural heritage, and values. Our results on
various understanding, translation, and cultural and values alignment
benchmarks show that NileChat outperforms existing Arabic-aware LLMs of similar
size and performs on par with larger models. We share our methods, data, and
models with the community to promote the inclusion and coverage of more diverse
communities in LLM development.
6835043630e39ad8b0fa3f01
Abdellah El Mekki
6835043630e39ad8b0fa3f02
Houdaifa Atou
6835043630e39ad8b0fa3f03
Omer Nacar
claimed_verified
2025-05-29T07:44:30.737Z
628f7a71dd993507cfcbe587
https://cdn-avatars.huggingface.co/v1/production/uploads/628f7a71dd993507cfcbe587/3frAcMgfBCx-OJsrdAhnb.png
Omartificial Intelligence Space
user
Omartificial-Intelligence-Space
6835043630e39ad8b0fa3f04
Shady Shehata
6835043630e39ad8b0fa3f05
Muhammad Abdul-Mageed
NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for
  Local Communities
Enhancing the linguistic capabilities of Large Language Models (LLMs) to
include low-resource languages is a critical research area. Current research
directions predominantly rely on synthetic data generated by translating
English corpora, which, while demonstrating promising linguistic understanding
and translation abilities, often results in models aligned with source language
culture. These models frequently fail to represent the cultural heritage and
values of local communities. This work proposes a methodology to create both
synthetic and retrieval-based pre-training data tailored to a specific
community, considering its (i) language, (ii) cultural heritage, and (iii)
cultural values. We demonstrate our methodology using Egyptian and Moroccan
dialects as testbeds, chosen for their linguistic and cultural richness and
current underrepresentation in LLMs. As a proof-of-concept, we develop
NileChat, a 3B parameter LLM adapted for Egyptian and Moroccan communities,
incorporating their language, cultural heritage, and values. Our results on
various understanding, translation, and cultural and values alignment
benchmarks show that NileChat outperforms existing Arabic-aware LLMs of similar
size and performs on par with larger models. We share our methods, data, and
models with the community to promote the inclusion and coverage of more diverse
communities in LLM development.
A methodology is proposed to create pre-training data tailored to low-resource languages and cultures, demonstrated through NileChat, a 3B parameter LLM for Egyptian and Moroccan dialects, which outperforms existing similar-sized Arabic-aware models.
Large Language Models synthetic data retrieval-based pre-training cultural heritage cultural values NileChat