readme: ---
  tags:
  - Transformers
  - text-classification
  - multi-class-classification
  languages:
  - af-ZA
  - am-ET
  - ar-SA
  - az-AZ
  - bn-BD
  - cy-GB
  - da-DK
  - de-DE
  - el-GR
  - en-US
  - es-ES
  - fa-IR
  - fi-FI
  - fr-FR
  - he-IL
  - hi-IN
  - hu-HU
  - hy-AM
  - id-ID
  - is-IS
  - it-IT
  - ja-JP
  - jv-ID
  - ka-GE
  - km-KH
  - kn-IN
  - ko-KR
  - lv-LV
  - ml-IN
  - mn-MN
  - ms-MY
  - my-MM
  - nb-NO
  - nl-NL
  - pl-PL
  - pt-PT
  - ro-RO
  - ru-RU
  - sl-SL
  - sq-AL
  - sv-SE
  - sw-KE
  - ta-IN
  - te-IN
  - th-TH
  - tl-PH
  - tr-TR
  - ur-PK
  - vi-VN
  - zh-CN
  - zh-TW
  multilinguality:
  - af-ZA
  - am-ET
  - ar-SA
  - az-AZ
  - bn-BD
  - cy-GB
  - da-DK
  - de-DE
  - el-GR
  - en-US
  - es-ES
  - fa-IR
  - fi-FI
  - fr-FR
  - he-IL
  - hi-IN
  - hu-HU
  - hy-AM
  - id-ID
  - is-IS
  - it-IT
  - ja-JP
  - jv-ID
  - ka-GE
  - km-KH
  - kn-IN
  - ko-KR
  - lv-LV
  - ml-IN
  - mn-MN
  - ms-MY
  - my-MM
  - nb-NO
  - nl-NL
  - pl-PL
  - pt-PT
  - ro-RO
  - ru-RU
  - sl-SL
  - sq-AL
  - sv-SE
  - sw-KE
  - ta-IN
  - te-IN
  - th-TH
  - tl-PH
  - tr-TR
  - ur-PK
  - vi-VN
  - zh-CN
  - zh-TW
  datasets:
  - qanastek/MASSIVE
  widget:
  - text: "wake me up at five am this week"
  - text: "je veux écouter la chanson de jacques brel encore une fois"
  - text: "quiero escuchar la canción de arijit singh una vez más"
  - text: "olly onde é que á um parque por perto onde eu possa correr"
  - text: "פרק הבא בפודקאסט בבקשה"
  - text: "亚马逊股价"
  - text: "найди билет на поезд в санкт-петербург"
  license: cc-by-4.0
  ---
  
  **People Involved**
  
  * [LABRAK Yanis](https://www.linkedin.com/in/yanis-labrak-8a7412145/) (1)
  
  **Affiliations**
  
  1. [LIA, NLP team](https://lia.univ-avignon.fr/), Avignon University, Avignon, France.
  
  ## Model
  
  XLM-Roberta : [https://huggingface.co/xlm-roberta-base](https://huggingface.co/xlm-roberta-base)
  
  Paper : [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/pdf/1911.02116.pdf)
  
  ## Demo: How to use in HuggingFace Transformers Pipeline
  
  Requires [transformers](https://pypi.org/project/transformers/): ```pip install transformers```
  
  ```python
  from transformers import AutoTokenizer, AutoModelForSequenceClassification, TextClassificationPipeline
  model_name = 'qanastek/51-languages-classifier'
  tokenizer = AutoTokenizer.from_pretrained(model_name)
  model = AutoModelForSequenceClassification.from_pretrained(model_name)
  classifier = TextClassificationPipeline(model=model, tokenizer=tokenizer)
  res = classifier("פרק הבא בפודקאסט בבקשה")
  print(res)
  ```
  
  Outputs:
  
  ```python
  [{'label': 'he-IL', 'score': 0.9998375177383423}]
  ```
  
  ## Training data
  
  [MASSIVE](https://huggingface.co/datasets/qanastek/MASSIVE) is a parallel dataset of > 1M utterances across 51 languages with annotations for the Natural Language Understanding tasks of intent prediction and slot annotation. Utterances span 60 intents and include 55 slot types. MASSIVE was created by localizing the SLURP dataset, composed of general Intelligent Voice Assistant single-shot interactions.
  
  ### Languages
  
  Thee model is capable of distinguish 51 languages :
  
  - `Afrikaans - South Africa (af-ZA)`
  - `Amharic - Ethiopia (am-ET)`
  - `Arabic - Saudi Arabia (ar-SA)`
  - `Azeri - Azerbaijan (az-AZ)`
  - `Bengali - Bangladesh (bn-BD)`
  - `Chinese - China (zh-CN)`
  - `Chinese - Taiwan (zh-TW)`
  - `Danish - Denmark (da-DK)`
  - `German - Germany (de-DE)`
  - `Greek - Greece (el-GR)`
  - `English - United States (en-US)`
  - `Spanish - Spain (es-ES)`
  - `Farsi - Iran (fa-IR)`
  - `Finnish - Finland (fi-FI)`
  - `French - France (fr-FR)`
  - `Hebrew - Israel (he-IL)`
  - `Hungarian - Hungary (hu-HU)`
  - `Armenian - Armenia (hy-AM)`
  - `Indonesian - Indonesia (id-ID)`
  - `Icelandic - Iceland (is-IS)`
  - `Italian - Italy (it-IT)`
  - `Japanese - Japan (ja-JP)`
  - `Javanese - Indonesia (jv-ID)`
  - `Georgian - Georgia (ka-GE)`
  - `Khmer - Cambodia (km-KH)`
  - `Korean - Korea (ko-KR)`
  - `Latvian - Latvia (lv-LV)`
  - `Mongolian - Mongolia (mn-MN)`
  - `Malay - Malaysia (ms-MY)`
  - `Burmese - Myanmar (my-MM)`
  - `Norwegian - Norway (nb-NO)`
  - `Dutch - Netherlands (nl-NL)`
  - `Polish - Poland (pl-PL)`
  - `Portuguese - Portugal (pt-PT)`
  - `Romanian - Romania (ro-RO)`
  - `Russian - Russia (ru-RU)`
  - `Slovanian - Slovania (sl-SL)`
  - `Albanian - Albania (sq-AL)`
  - `Swedish - Sweden (sv-SE)`
  - `Swahili - Kenya (sw-KE)`
  - `Hindi - India (hi-IN)`
  - `Kannada - India (kn-IN)`
  - `Malayalam - India (ml-IN)`
  - `Tamil - India (ta-IN)`
  - `Telugu - India (te-IN)`
  - `Thai - Thailand (th-TH)`
  - `Tagalog - Philippines (tl-PH)`
  - `Turkish - Turkey (tr-TR)`
  - `Urdu - Pakistan (ur-PK)`
  - `Vietnamese - Vietnam (vi-VN)`
  - `Welsh - United Kingdom (cy-GB)`
  
  ## Evaluation results
  
  ```plain
                precision    recall  f1-score   support
  
         af-ZA     0.9821    0.9805    0.9813      2974
         am-ET     1.0000    1.0000    1.0000      2974
         ar-SA     0.9809    0.9822    0.9815      2974
         az-AZ     0.9946    0.9845    0.9895      2974
         bn-BD     0.9997    0.9990    0.9993      2974
         cy-GB     0.9970    0.9929    0.9949      2974
         da-DK     0.9575    0.9617    0.9596      2974
         de-DE     0.9906    0.9909    0.9908      2974
         el-GR     0.9997    0.9973    0.9985      2974
  