readme: ---
  language: 
  - ar
  license: apache-2.0
  widget:
   - text: "الهدف من الحياة هو [MASK] ."
  ---
  
  # CAMeLBERT: A collection of pre-trained models for Arabic NLP tasks
  
  ## Model description
  
  **CAMeLBERT** is a collection of BERT models pre-trained on Arabic texts with different sizes and variants.
  We release pre-trained language models for Modern Standard Arabic (MSA), dialectal Arabic (DA), and classical Arabic (CA), in addition to a model pre-trained on a mix of the three.
  We also provide additional models that are pre-trained on a scaled-down set of the MSA variant (half, quarter, eighth, and sixteenth).
  The details are described in the paper *"[The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://arxiv.org/abs/2103.06678)."*
  
  This model card describes **CAMeLBERT-MSA-sixteenth** (`bert-base-arabic-camelbert-msa-sixteenth`), a model pre-trained on a sixteenth of the full MSA dataset.
  
  ||Model|Variant|Size|#Word|
  |-|-|:-:|-:|-:|
  ||`bert-base-arabic-camelbert-mix`|CA,DA,MSA|167GB|17.3B|
  ||`bert-base-arabic-camelbert-ca`|CA|6GB|847M|
  ||`bert-base-arabic-camelbert-da`|DA|54GB|5.8B|
  ||`bert-base-arabic-camelbert-msa`|MSA|107GB|12.6B|
  ||`bert-base-arabic-camelbert-msa-half`|MSA|53GB|6.3B|
  ||`bert-base-arabic-camelbert-msa-quarter`|MSA|27GB|3.1B|
  ||`bert-base-arabic-camelbert-msa-eighth`|MSA|14GB|1.6B|
  |✔|`bert-base-arabic-camelbert-msa-sixteenth`|MSA|6GB|746M|
  
  ## Intended uses
  You can use the released model for either masked language modeling or next sentence prediction.
  However, it is mostly intended to be fine-tuned on an NLP task, such as NER, POS tagging, sentiment analysis, dialect identification, and poetry classification.
  We release our fine-tuninig code [here](https://github.com/CAMeL-Lab/CAMeLBERT).
  
  #### How to use
  You can use this model directly with a pipeline for masked language modeling:
  ```python
  >>> from transformers import pipeline
  >>> unmasker = pipeline('fill-mask', model='CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
  >>> unmasker("الهدف من الحياة هو [MASK] .")
  [{'sequence': '[CLS] الهدف من الحياة هو التغيير. [SEP]',
    'score': 0.08320745080709457,
    'token': 7946,
    'token_str': 'التغيير'},
   {'sequence': '[CLS] الهدف من الحياة هو التعلم. [SEP]',
    'score': 0.04305094853043556,
    'token': 12554,
    'token_str': 'التعلم'},
   {'sequence': '[CLS] الهدف من الحياة هو العمل. [SEP]',
    'score': 0.0417640283703804,
    'token': 2854,
    'token_str': 'العمل'},
   {'sequence': '[CLS] الهدف من الحياة هو الحياة. [SEP]',
    'score': 0.041371218860149384,
    'token': 3696,
    'token_str': 'الحياة'},
   {'sequence': '[CLS] الهدف من الحياة هو المعرفة. [SEP]',
    'score': 0.039794355630874634,
    'token': 7344,
    'token_str': 'المعرفة'}]
  ```
  
  *Note*: to download our models, you would need `transformers>=3.5.0`. Otherwise, you could download the models manually.
  
  Here is how to use this model to get the features of a given text in PyTorch:
  ```python
  from transformers import AutoTokenizer, AutoModel
  tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
  model = AutoModel.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
  text = "مرحبا يا عالم."
  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)
  ```
  
  and in TensorFlow:
  ```python
  from transformers import AutoTokenizer, TFAutoModel
  tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
  model = TFAutoModel.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
  text = "مرحبا يا عالم."
  encoded_input = tokenizer(text, return_tensors='tf')
  output = model(encoded_input)
  ```
  
  ## Training data
  - MSA (Modern Standard Arabic)
    - [The Arabic Gigaword Fifth Edition](https://catalog.ldc.upenn.edu/LDC2011T11)
    - [Abu El-Khair Corpus](http://www.abuelkhair.net/index.php/en/arabic/abu-el-khair-corpus)
    - [OSIAN corpus](https://vlo.clarin.eu/search;jsessionid=31066390B2C9E8C6304845BA79869AC1?1&q=osian)
    - [Arabic Wikipedia](https://archive.org/details/arwiki-20190201)
    - The unshuffled version of the Arabic [OSCAR corpus](https://oscar-corpus.com/)
  
  ## Training procedure
  We use [the original implementation](https://github.com/google-research/bert) released by Google for pre-training.
  We follow the original English BERT model's hyperparameters for pre-training, unless otherwise specified.
  
  ### Preprocessing
  - After extracting the raw text from each corpus, we apply the following pre-processing.
  - We first remove invalid characters and normalize white spaces using the utilities provided by [the original BERT implementation](https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/tokenization.py#L286-L297).
  - We also remove lines without any Arabic characters.
  - We then remove diacritics and kashida using [CAMeL Tools](https://github.com/CAMeL-Lab/camel_tools).
  - Finally, we split each line into sentences with a heuristics-based sentence segmenter.
  - We train a WordPiece tok