readme: ---
  license: openrail
  language:
  - dan
  - eng
  - fao
  - fin
  - isl
  - nno
  - nob
  - sma
  - sme
  - smj
  - smn
  - sms
  - swe
  tags:
  - fasttext
  - text-classification
  - language-detection
  - language-identification
  datasets:
  - tatoeba
  library_name: fasttext
  inference: false
  ---
  
  # NB-NORDIC-LID
  
  This repo contains models for the identification of language in text (also referred to as language detection). It is based on Fasttext and designed with the Nordic languages in mind, including several Sámi languages. It comes in two flavours, `nb-nordic-lid`, a model that identifies between the 12 most common languages in the Nordic countries (plus English), and `nb-nordic-lid.159`, a model that extends that list to 159 languages of the world. Moreover, each of them come in large and small (quantized) versions.
  
  | Model                       | Size              |   Precision |   Recall |   F1-Score |   Support |
  |:----------------------------|:------------------|------------:|---------:|-----------:|----------:|
  | [`nb-nordic-lid.bin`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.bin) (large)    | 274 MB            |      0.9901 |   0.9900 |     0.9900 |      5500 |
  | [`nb-nordic-lid.ftz`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.ftz) (small)    | 1.87 MB           |      0.9889 |   0.9890 |     0.9890 |      5500 |
  | [`nb-nordic-lid.159.bin`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.159.bin) (large)| 9.63 GB           |      0.9434 |   0.9528 |     0.9476 |     44049 |
  | [`nb-nordic-lid.159.ftz`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.159.ftz) (small)| 11.2 MB           |      0.9275 |   0.9399 |     0.9327 |     44049 |
  
  
  ## Usage
  
  After download, the models can be used through the Fasttext library:
  
  ```python
  import fasttext
  from datasets.utils.download_manager import DownloadManager
  
  
  NORDIC_LID_URL = "https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/"
  model_name = "nb-nordic-lid.ftz"
  
  model = fasttext.load_model(DownloadManager().download(NORDIC_LID_URL + model_name))
  model.predict("Debatt er bra og sunt for demokratier, og en forutsetning for politikkutvikling.", threshold=0.25)
  # (('__label__nob',), array([0.95482141]))
  ```
  
  Alternatively, these models are also integrated into the the experimental `nbailab` CLI application:
  
  ```bash
  $ echo "Jeg leser en bok" | nbailab langid --model-name  nb-nordic-lid.ftz
  nob,0.9999788999557495
  ```
  
  
  ## Languages
  
  ### `nb-nordic-lid.bin`
  
  Trained on sentences from the [GiellaT's Tranlation Memories](https://giellalt.github.io/tm/TranslationMemories.html) and [Wortschatz's corpora](https://wortschatz.uni-leipzig.de/en/download).
  
  | ISO-639-3    | Language          |   Precision |   Recall |   F1-Score |   Support |
  |:-------------|:------------------|------------:|---------:|-----------:|----------:|
  | dan          | Danish            |      0.9720 |   0.9838 |     0.9779 |       494 |
  | eng          | English           |      0.9980 |   0.9940 |     0.9960 |       502 |
  | fao          | Faroese           |      0.9920 |   0.9940 |     0.9930 |       499 |
  | fin          | Finnish           |      1.0000 |   1.0000 |     1.0000 |       500 |
  | isl          | Icelandic         |      0.9900 |   0.9920 |     0.9910 |       499 |
  | nno          | Norwegian Nynorsk |      0.9920 |   0.9861 |     0.9890 |       503 |
  | nob          | Norwegian Bokmål  |      0.9840 |   0.9743 |     0.9791 |       505 |
  | sma          | Southern Sami     |      0.9800 |   0.9703 |     0.9751 |       101 |
  | sme          | Northern Sami     |      1.0000 |   0.9921 |     0.9960 |       504 |
  | smj          | Lule Sami         |      0.9920 |   0.9960 |     0.9940 |       498 |
  | smn          | Inari Sami        |      0.9950 |   1.0000 |     0.9975 |       199 |
  | sms          | Skolt Sami        |      0.9900 |   0.9950 |     0.9925 |       199 |
  | swe          | Swedish           |      0.9860 |   0.9920 |     0.9890 |       497 |
  | Accuracy     |                   |             |          |     0.9905 |      5500 |
  | Weighted avg |                   |      0.9906 |   0.9905 |     0.9905 |      5500 |
  | Macro avg    |                   |      0.9901 |   0.9900 |     0.9900 |      5500 |
  
  ### `nb-nordic-lid.159.bin`
  
  <details>
    <summary>Scores for the 159 languages</summary>
    
  Additionally trained on sentences from [Taoteba](https://tatoeba.org/en/).
  
  | ISO-639-3    | Language                    |   Precision |   Recall |   F1-Score |   Support |
  |:-------------|:----------------------------|------------:|---------:|-----------:|----------:|
  | afr          | Afrikaans                   |      0.9634 |   0.9485 |     0.9558 |       194 |
  | ara          | Arabic                      |      0.9771 |   0.9533 |     0.9650 |       492 |
  | arq          | Algerian Arabic             |      0.9478 |   0.9316 |     0.9397 |       117 |
  | arz          | Egyptian Arabic             |      0.7193 |