transformers pytorch tensorboard roberta fill-mask generated_from_trainer ar dataset:SaiedAlshahrani/Moroccan_Arabic_Wikipedia_20230101_bots dataset:SaiedAlshahrani/MASD license:mit autotrain_compatible endpoints_compatible region:us
---
datasets:
- SaiedAlshahrani/Moroccan_Arabic_Wikipedia_20230101_bots
- SaiedAlshahrani/MASD
language:
- ar
library_name: transformers
license: mit
metrics:
- perplexity
pipeline_tag: fill-mask
tags:
- generated_from_trainer
widget:
- text: الهدف من الحياة هو  <mask>
model-index:
- name: aryRoBERTa
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Moroccan Arabic Wikipedia (aryRoBERTa<sub>BASE</sub>)

This aryRoBERTa<sub>BASE</sub> model has been trained *from scratch* on the Moroccan Arabic Wikipedia articles, downloaded on the 1st of January 2023, processed using
`Gensim` Python library, preprocessed using `tr` Linux/Unix utility and `CAMeLTools` Python toolkit for Arabic NLP, and hosted here at [SaiedAlshahrani/Moroccan\_Arabic\_Wikipedia\_20230101\_bots](https://huggingface.co/datasets/SaiedAlshahrani/Moroccan_Arabic_Wikipedia_20230101_bots).
It achieves the following results on the evaluation set:

- Pseudo-Perplexity: 5,379.89

## Model description

We trained this Moroccan Arabic Wikipedia Masked Language Model (aryRoBERTa<sub>BASE</sub>) to evaluate its performance using the Fill-Mask evaluation task and the Masked Arab States Dataset ([MASD](https://huggingface.co/datasets/SaiedAlshahrani/MASD)) dataset and measure the *impact* of **template-based translation** on the Egyptian Arabic Wikipedia edition.

For more details about the experiment, please **read** and **cite** our paper:

```bash
@inproceedings{alshahrani-etal-2023-performance,
    title = "{Performance Implications of Using Unrepresentative Corpora in {A}rabic Natural Language Processing}",
    author = "Alshahrani, Saied  and Alshahrani, Norah  and Dey, Soumyabrata  and Matthews, Jeanna",
    booktitle = "Proceedings of the The First Arabic Natural Language Processing Conference (ArabicNLP 2023)",
    month = December,
    year = "2023",
    address = "Singapore (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.arabicnlp-1.19",
    doi = "10.18653/v1/2023.arabicnlp-1.19",
    pages = "218--231",
    abstract = "Wikipedia articles are a widely used source of training data for Natural Language Processing (NLP) research, particularly as corpora for low-resource languages like Arabic. However, it is essential to understand the extent to which these corpora reflect the representative contributions of native speakers, especially when many entries in a given language are directly translated from other languages or automatically generated through automated mechanisms. In this paper, we study the performance implications of using inorganic corpora that are not representative of native speakers and are generated through automated techniques such as bot generation or automated template-based translation. The case of the Arabic Wikipedia editions gives a unique case study of this since the Moroccan Arabic Wikipedia edition (ARY) is small but representative, the Egyptian Arabic Wikipedia edition (ARZ) is large but unrepresentative, and the Modern Standard Arabic Wikipedia edition (AR) is both large and more representative. We intrinsically evaluate the performance of two main NLP upstream tasks, namely word representation and language modeling, using word analogy evaluations and fill-mask evaluations using our two newly created datasets: Arab States Analogy Dataset (ASAD) and Masked Arab States Dataset (MASD). We demonstrate that for good NLP performance, we need both large and organic corpora; neither alone is sufficient. We show that producing large corpora through automated means can be a counter-productive, producing models that both perform worse and lack cultural richness and meaningful representation of the Arabic language and its native speakers.",
}
```

## Intended uses & limitations

We do **not** recommend using this model because it was trained *only* on the Moroccan Arabic Wikipedia articles, <u>unless</u> you fine-tune the model on a large, organic, and representative Moroccan Arabic dataset.

## Training and evaluation data

We have trained this model on the Moroccan Arabic Wikipedia articles ([SaiedAlshahrani/Moroccan\_Arabic\_Wikipedia\_20230101\_bots](https://huggingface.co/datasets/SaiedAlshahrani/Moroccan_Arabic_Wikipedia_20230101_bots)) without using any validation or evaluation data (only training data) due to a lack of computational power.

## Training procedure

We have trained this model using the Paperspace GPU-Cloud service. We used a machine with 8 CPUs, 45GB RAM, and A6000 GPU with 48GB RAM.

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 128
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-06
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Epoch | Step  | Training Loss | 
|:-----:|:-----:|:-------------:|
| 1     | 35    | 9.561500      | 
| 2     | 70    | 7.946000      | 
| 3     | 105   | 7.420400      | 
| 4     | 140   | 7.197800      |
| 5     | 175   | 7.174400      | 

| Train Runtime  | Train Samples Per Second | Train Steps Per Second | Total Flos                | Train Loss |  Epoch   |
|:--------------:|:------------------------:|:----------------------:|:-------------------------:|:----------:|:--------:|
| 192.684800     | 121.260000              | 0.960000                | 774708261150720.000000     | 7.812142   | 5.000000 |

### Evaluation results
This aryRoBERTa<sub>BASE</sub> model has been evaluated on the Masked Arab States Dataset ([SaiedAlshahrani/MASD](https://huggingface.co/datasets/SaiedAlshahrani/MASD)).
| K=10 | K=50  | K=100 | 
|:----:|:-----:|:----:|
| 0.00%| 0.00% | 0.62% | 

### Framework versions

- Datasets 2.9.0
- Tokenizers 0.12.1
- Transformers 4.24.0
- Pytorch 1.12.1+cu116