README.md

metadata

base_model: EuroBERT/EuroBERT-210m
language:
  - ar
library_name: sentence-transformers
license: mit
metrics:
  - pearson_cosine
  - spearman_cosine
pipeline_tag: feature-extraction
tags:
  - sentence-transformers
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
  - EuroBert
  - Arabic
widget:
  - source_sentence: امرأة شقراء تطل على مشهد (سياتل سبيس نيدل)
    sentences:
      - رجل يستمتع بمناظر جسر البوابة الذهبية
      - فتاة بالخارج تلعب في الثلج
      - شخص ما يأخذ في نظرة إبرة الفضاء.
  - source_sentence: سوق الشرق الأوسط
    sentences:
      - مسرح أمريكي
      - متجر في الشرق الأوسط
      - البالغون صغار
  - source_sentence: رجلين يتنافسان في ملابس فنون الدفاع عن النفس
    sentences:
      - هناك العديد من الناس الحاضرين.
      - الكلب الأبيض على الشاطئ
      - هناك شخص واحد فقط موجود.\
  - source_sentence: مجموعة من الناس تمشي بجانب شاحنة.
    sentences:
      - الناس يقفون
      - بعض الناس بالخارج
      - بعض الرجال يقودون على الطريق
  - source_sentence: لاعبة كرة ناعمة ترمي الكرة إلى زميلتها في الفريق
    sentences:
      - شخصان يلعبان كرة البيسبول
      - الرجل ينظف
      - لاعبين لكرة البيسبول يجلسان على مقعد
model-index:
  - name: SentenceTransformer based on EuroBERT/EuroBERT-210m
    results:
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: sts dev 768
          type: sts-dev-768
        metrics:
          - type: pearson_cosine
            value: 0.8111988062913815
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.8100586279907306
            name: Spearman Cosine
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: sts dev 512
          type: sts-dev-512
        metrics:
          - type: pearson_cosine
            value: 0.8092891955563192
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.8087644228771842
            name: Spearman Cosine
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: sts dev 256
          type: sts-dev-256
        metrics:
          - type: pearson_cosine
            value: 0.8076510620939634
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.8080588277305082
            name: Spearman Cosine
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: sts dev 128
          type: sts-dev-128
        metrics:
          - type: pearson_cosine
            value: 0.8028710019029521
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.8054855987917489
            name: Spearman Cosine
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: sts dev 64
          type: sts-dev-64
        metrics:
          - type: pearson_cosine
            value: 0.7923252906438638
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.7975941111911333
            name: Spearman Cosine

Ara-EuroBERT: Arabic-optimized Sentence Transformer

Ara-EuroBERT is a sentence-transformers model fine-tuned from EuroBERT/EuroBERT-210m specifically optimized for Semantic Arabic text embeddings.

This model maps sentences and paragraphs to a 768-dimensional dense vector space and Maximum Sequence Length: 8,192 tokens.

Paper:

You can find more information on the base model at https://huggingface.co/EuroBERT

Our fine-tuned model shows remarkable improvements over the base models, achieving a 73.5% relative improvement on STS17 and a 21.6% relative improvement on STS22.v2 compared to the base EuroBERT-210M.

Model Details

Model Description

• Model Type: Sentence Transformer with Matryoshka Embeddings
• Base model: EuroBERT/EuroBERT-210m
• Maximum Sequence Length: 8,192 tokens
• Output Dimensionality: Matryoshka embeddings with dimensions [768, 512, 256, 128, 64]
• Similarity Function: Cosine Similarity
• Languages: Optimized for Arabic
• License: Same as EuroBERT (MIT)

Matryoshka Embeddings

This model is trained with Matryoshka Representation Learning, allowing for flexible embedding dimensionality without retraining. You can use smaller dimensions (64, 128, 256, 512) for efficiency or the full 768 dimensions for maximum performance. The model maintains strong performance even at reduced dimensions:

Dimension	Spearman Correlation (STS Dev)
768	0.8101
512	0.8088
256	0.8081
128	0.8055
64	0.7976

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: EuroBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Use Cases

This model excels at various Arabic NLP tasks:

• Semantic textual similarity
• Semantic search and information retrieval
• Document clustering and classification
• Question answering
• Paraphrase detection
• Zero-shot classification

Usage

Installation

pip install -U sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer("Omartificial-Intelligence-Space/AraEuroBert-210M")

# Encode Arabic sentences
sentences = [
    'التقدم العلمي في مجال الذكاء الاصطناعي يتسارع بشكل ملحوظ في السنوات الأخيرة',
    'تطوير نماذج لغوية متقدمة يساهم في تحسين فهم اللغة العربية آليًا',
    'أصبحت تقنيات معالجة اللغات الطبيعية جزءًا أساسيًا من التطبيقات الحديثة',
    'يعاني الشرق الأوسط من تحديات مناخية متزايدة تهدد الأمن المائي والغذائي',
    'تراث الأدب العربي غني بالقصائد والروايات التي تعكس تاريخ وثقافة المنطقة',
]
embeddings = model.encode(sentences)
print(embeddings.shape)  # [3, 768]

# Get the similarity scores
from sentence_transformers import util
similarities = util.cos_sim(embeddings, embeddings)
print(similarities)

Using Matryoshka Embeddings

# Get embeddings with different dimensions
embeddings_768 = model.encode(sentences)  # Default: full 768 dimensions
embeddings_256 = model.encode(sentences, truncate_dim=256)  # Use only 256 dimensions
embeddings_64 = model.encode(sentences, truncate_dim=64)  # Use only 64 dimensions

Training Method

• Loss Function: MatryoshkaLoss with MultipleNegativesRankingLoss
• Matryoshka Dimensions: [768, 512, 256, 128, 64]
• Batch Size: 32
• Epochs: 1 (with early stopping)
• Optimizer: AdamW
• Learning Rate: 5e-05 with linear scheduler and 10% warmup
• Hardware: Multiple NVIDIA GPUs with mixed precision (fp16)

Base Model: EuroBERT

EuroBERT is a new family of multilingual encoder models designed specifically for European and widely spoken global languages. It offers several advantages over traditional multilingual encoders:

• Extensive Multilingual Coverage: Trained on a 5 trillion-token dataset across 15 languages
• Advanced Architecture: Uses grouped query attention, rotary position embeddings, and RMS normalization
• Long Context Support: Natively processes up to 8,192 tokens
• Specialized Knowledge: Includes math and programming language data for improved reasoning

Limitations and Recommendations

• The model is primarily optimized for Arabic text and may not perform optimally on other languages
• Performance may vary for specialized domains not well-represented in the training data
• For short texts (<5 words), consider augmenting with context for better representations
• For extremely long documents, consider splitting into meaningful chunks before encoding

Citation

If you use this model in your research, please cite the following works:

@misc{boizard2025eurobertscalingmultilingualencoders,
      title={EuroBERT: Scaling Multilingual Encoders for European Languages}, 
      author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves and André Martins and Ayoub Hammal and Caio Corro and Céline Hudelot and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan and Gabriel Hautreux and João Alves and Kevin El-Haddad and Manuel Faysse and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes and Ricardo Rei and Pierre Colombo},\
      year={2025},\
      eprint={2503.05500},\
      archivePrefix={arXiv},\
      primaryClass={cs.CL},\
      url={https://arxiv.org/abs/2503.05500}, 
}

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},\
    eprint={2205.13147},\
    archivePrefix={arXiv},\
    primaryClass={cs.LG}\
}