readme: ---
  language: wo
  tags:
  - bert
  - language-model
  - wo
  - wolof 
  ---
  
  # Soraberta: Unsupervised Language Model Pre-training for Wolof
  
  **bert-base-wolof** is pretrained bert-base model on wolof language  .
  
  ## Soraberta models
  
  | Model name | Number of layers | Attention Heads | Embedding Dimension | Total Parameters |
  | :------:       |   :---: | :---: | :---: | :---: |
  | `bert-base` | 6    | 12   | 514   | 56931622 M |
   
  
  
  
  ## Using Soraberta with Hugging Face's Transformers
  
  
  ```python
  >>> from transformers import pipeline
  >>> unmasker = pipeline('fill-mask', model='abdouaziiz/bert-base-wolof')
  >>> unmasker("kuy yoot du [MASK].")
  
  [{'sequence': '[CLS] kuy yoot du seqet. [SEP]',
    'score': 0.09505125880241394,
    'token': 13578},
   {'sequence': '[CLS] kuy yoot du daw. [SEP]',
    'score': 0.08882280439138412,
    'token': 679},
   {'sequence': '[CLS] kuy yoot du yoot. [SEP]',
    'score': 0.057790059596300125,
    'token': 5117},
   {'sequence': '[CLS] kuy yoot du seqat. [SEP]',
    'score': 0.05671025067567825,
    'token': 4992},
   {'sequence': '[CLS] kuy yoot du yaqu. [SEP]',
    'score': 0.0469999685883522,
    'token': 1735}]
  ```
  
  ## Training data
  The data sources are [Bible OT](http://biblewolof.com/) , [WOLOF-ONLINE](http://www.wolof-online.com/) 
  [ALFFA_PUBLIC](https://github.com/getalp/ALFFA_PUBLIC/tree/master/ASR/WOLOF)
  
  
  
  ## Contact
  
  Please contact abdouaziz@gmail.com for any question, feedback or request.