fasttext text-classification language-identification language-detection dan eng fao fin isl nno nob sma sme smj smn sms swe dataset:tatoeba license:openrail region:us
---
datasets:
- tatoeba
language:
- dan
- eng
- fao
- fin
- isl
- nno
- nob
- sma
- sme
- smj
- smn
- sms
- swe
library_name: fasttext
license: openrail
tags:
- text-classification
- language-identification
- language-detection
---

# NB-NORDIC-LID

This repo contains models for the identification of language in text (also referred to as language detection). It is based on Fasttext and designed with the Nordic languages in mind, including several Sámi languages. It comes in two flavours, `nb-nordic-lid`, a model that identifies between the 12 most common languages in the Nordic countries (plus English), and `nb-nordic-lid.159`, a model that extends that list to 159 languages of the world. Moreover, each of them come in large and small (quantized) versions.

| Model                       | Size              |   Precision |   Recall |   F1-Score |   Support |
|:----------------------------|:------------------|------------:|---------:|-----------:|----------:|
| [`nb-nordic-lid.bin`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.bin) (large)    | 274 MB            |      0.9901 |   0.9900 |     0.9900 |      5500 |
| [`nb-nordic-lid.ftz`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.ftz) (small)    | 1.87 MB           |      0.9889 |   0.9890 |     0.9890 |      5500 |
| [`nb-nordic-lid.159.bin`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.159.bin) (large)| 9.63 GB           |      0.9434 |   0.9528 |     0.9476 |     44049 |
| [`nb-nordic-lid.159.ftz`](https://huggingface.co/NbAiLab/nb-nordic-lid/resolve/main/nb-nordic-lid.159.ftz) (small)| 11.2 MB           |      0.9275 |   0.9399 |     0.9327 |     44049 |


## Usage

After download, the models can be used through the Fasttext library:

```python
from huggingface_hub import hf_hub_download
import fasttext

model_name = "nb-nordic-lid.ftz"
model = fasttext.load_model(hf_hub_download("NbAiLab/nb-nordic-lid", model_name))

model.predict("Debatt er bra og sunt for demokratier, og en forutsetning for politikkutvikling.", threshold=0.25)
# (('__label__nob',), array([0.95482141]))
```

Alternatively, these models are also integrated into the the experimental `nbailab` CLI application:

```bash
$ echo "Jeg leser en bok" | nbailab langid --model-name  nb-nordic-lid.ftz
nob,0.9999788999557495
```


## Languages

### `nb-nordic-lid.bin`

Trained on sentences from the [GiellaT's Translation Memories](https://giellalt.github.io/tm/TranslationMemories.html) and [Wortschatz's corpora](https://wortschatz.uni-leipzig.de/en/download).

| ISO-639-3    | Language          |   Precision |   Recall |   F1-Score |   Support |
|:-------------|:------------------|------------:|---------:|-----------:|----------:|
| dan          | Danish            |      0.9720 |   0.9838 |     0.9779 |       494 |
| eng          | English           |      0.9980 |   0.9940 |     0.9960 |       502 |
| fao          | Faroese           |      0.9920 |   0.9940 |     0.9930 |       499 |
| fin          | Finnish           |      1.0000 |   1.0000 |     1.0000 |       500 |
| isl          | Icelandic         |      0.9900 |   0.9920 |     0.9910 |       499 |
| nno          | Norwegian Nynorsk |      0.9920 |   0.9861 |     0.9890 |       503 |
| nob          | Norwegian Bokmål  |      0.9840 |   0.9743 |     0.9791 |       505 |
| sma          | Southern Sami     |      0.9800 |   0.9703 |     0.9751 |       101 |
| sme          | Northern Sami     |      1.0000 |   0.9921 |     0.9960 |       504 |
| smj          | Lule Sami         |      0.9920 |   0.9960 |     0.9940 |       498 |
| smn          | Inari Sami        |      0.9950 |   1.0000 |     0.9975 |       199 |
| sms          | Skolt Sami        |      0.9900 |   0.9950 |     0.9925 |       199 |
| swe          | Swedish           |      0.9860 |   0.9920 |     0.9890 |       497 |
| Accuracy     |                   |             |          |     0.9905 |      5500 |
| Weighted avg |                   |      0.9906 |   0.9905 |     0.9905 |      5500 |
| Macro avg    |                   |      0.9901 |   0.9900 |     0.9900 |      5500 |

### `nb-nordic-lid.159.bin`

<details>
  <summary>Scores for the 159 languages</summary>
  
Additionally trained on sentences from [Taoteba](https://tatoeba.org/en/).

| ISO-639-3    | Language                    |   Precision |   Recall |   F1-Score |   Support |
|:-------------|:----------------------------|------------:|---------:|-----------:|----------:|
| afr          | Afrikaans                   |      0.9634 |   0.9485 |     0.9558 |       194 |
| ara          | Arabic                      |      0.9771 |   0.9533 |     0.9650 |       492 |
| arq          | Algerian Arabic             |      0.9478 |   0.9316 |     0.9397 |       117 |
| arz          | Egyptian Arabic             |      0.7193 |   0.8542 |     0.7810 |        48 |
| asm          | Assamese                    |      0.9828 |   0.9884 |     0.9856 |       173 |
| avk          | Kotava                      |      0.9895 |   0.9844 |     0.9869 |       192 |
| aze          | Azerbaijani                 |      0.9707 |   0.9831 |     0.9768 |       236 |
| bel          | Belarusian                  |      0.9864 |   0.9785 |     0.9825 |       372 |
| ben          | Bengali                     |      0.9915 |   0.9873 |     0.9894 |       236 |
| ber          | Berber                      |      0.8991 |   0.8507 |     0.8742 |       576 |
| bos          | Bosnian                     |      0.1548 |   0.1781 |     0.1656 |        73 |
| bre          | Breton                      |      0.9613 |   0.9681 |     0.9647 |       282 |
| bua          | Buryat                      |      0.9333 |   0.9130 |     0.9231 |        46 |
| bul          | Bulgarian                   |      0.9530 |   0.9660 |     0.9595 |       441 |
| cat          | Catalan                     |      0.9604 |   0.9510 |     0.9557 |       306 |
| cbk          | Chavacano                   |      0.9627 |   0.9923 |     0.9773 |       130 |
| ceb          | Cebuano                     |      0.8974 |   0.9091 |     0.9032 |        77 |
| ces          | Czech                       |      0.9684 |   0.9665 |     0.9675 |       508 |
| chv          | Chuvash                     |      0.9878 |   0.9643 |     0.9759 |        84 |
| ckb          | Central Kurdish (Soranî)    |      0.9751 |   0.9944 |     0.9846 |       354 |
| ckt          | Chukchi                     |      0.9615 |   1.0000 |     0.9804 |        25 |
| cmn          | Mandarin Chinese            |      0.9726 |   0.8674 |     0.9170 |       573 |
| cor          | Cornish                     |      1.0000 |   0.9733 |     0.9864 |       187 |
| csb          | Kashubian                   |      0.9787 |   1.0000 |     0.9892 |        46 |
| cym          | Welsh                       |      0.9625 |   0.9625 |     0.9625 |        80 |
| dan          | Danish                      |      0.9401 |   0.9345 |     0.9373 |      1007 |
| deu          | German                      |      0.9908 |   0.9765 |     0.9836 |       553 |
| dsb          | Lower Sorbian               |      0.8704 |   0.8246 |     0.8468 |        57 |
| dtp          | Central Dusun               |      0.9161 |   0.9562 |     0.9357 |       137 |
| ell          | Greek                       |      1.0000 |   0.9979 |     0.9989 |       476 |
| eng          | English                     |      0.9914 |   0.9886 |     0.9900 |      1052 |
| epo          | Esperanto                   |      0.9817 |   0.9853 |     0.9835 |       544 |
| est          | Estonian                    |      0.9659 |   0.9770 |     0.9714 |       174 |
| eus          | Basque                      |      0.9883 |   0.9585 |     0.9732 |       265 |
| fao          | Faroese                     |      0.9840 |   0.9899 |     0.9870 |       497 |
| fin          | Finnish                     |      0.9932 |   0.9817 |     0.9874 |      1041 |
| fkv          | Kven Finnish                |      0.5769 |   0.7500 |     0.6522 |        20 |
| fra          | French                      |      0.9890 |   0.9890 |     0.9890 |       544 |
| frr          | North Frisian               |      0.9784 |   0.9784 |     0.9784 |       139 |
| fry          | Frisian                     |      0.7419 |   0.9200 |     0.8214 |        25 |
| gcf          | Guadeloupean Creole French  |      0.9810 |   0.9904 |     0.9856 |       104 |
| gla          | Scottish Gaelic             |      0.9608 |   0.9800 |     0.9703 |        50 |
| gle          | Irish                       |      0.9781 |   0.9853 |     0.9817 |       136 |
| glg          | Galician                    |      0.9198 |   0.9330 |     0.9264 |       209 |
| gos          | Gronings                    |      0.9631 |   0.9671 |     0.9651 |       243 |
| grc          | Ancient Greek               |      0.9828 |   1.0000 |     0.9913 |        57 |
| grn          | Guarani                     |      0.9810 |   0.9936 |     0.9873 |       156 |
| guc          | Wayuu                       |      0.9556 |   1.0000 |     0.9773 |        43 |
| hau          | Hausa                       |      0.9930 |   0.9930 |     0.9930 |       431 |
| heb          | Hebrew                      |      1.0000 |   1.0000 |     1.0000 |       536 |
| hin          | Hindi                       |      1.0000 |   0.9974 |     0.9987 |       391 |
| hoc          | Ho                          |      0.9143 |   1.0000 |     0.9552 |        32 |
| hrv          | Croatian                    |      0.6085 |   0.5652 |     0.5861 |       253 |
| hrx          | Hunsrik                     |      0.8727 |   0.9231 |     0.8972 |        52 |
| hsb          | Upper Sorbian               |      0.8533 |   0.8312 |     0.8421 |        77 |
| hun          | Hungarian                   |      0.9853 |   0.9889 |     0.9871 |       541 |
| hye          | Armenian                    |      1.0000 |   1.0000 |     1.0000 |       225 |
| ido          | Ido                         |      0.9731 |   0.9560 |     0.9645 |       341 |
| ile          | Interlingue                 |      0.9386 |   0.9450 |     0.9418 |       291 |
| ilo          | Ilocano                     |      0.9917 |   0.9677 |     0.9796 |       124 |
| ina          | Interlingua                 |      0.9602 |   0.9775 |     0.9688 |       444 |
| ind          | Indonesian                  |      0.8550 |   0.8305 |     0.8426 |       419 |
| isl          | Icelandic                   |      0.9874 |   0.9931 |     0.9902 |       869 |
| ita          | Italian                     |      0.9835 |   0.9746 |     0.9791 |       552 |
| jav          | Javanese                    |      0.9400 |   0.9792 |     0.9592 |        48 |
| jbo          | Lojban                      |      1.0000 |   1.0000 |     1.0000 |       402 |
| jpn          | Japanese                    |      0.9870 |   1.0000 |     0.9935 |       531 |
| kab          | Kabyle                      |      0.8382 |   0.9012 |     0.8686 |       506 |
| kat          | Georgian                    |      1.0000 |   0.9885 |     0.9942 |       260 |
| kaz          | Kazakh                      |      0.9896 |   0.9896 |     0.9896 |       192 |
| kha          | Khasi                       |      0.9038 |   0.9400 |     0.9216 |       100 |
| khm          | Khmer                       |      1.0000 |   1.0000 |     1.0000 |        75 |
| kmr          | Northern Kurdish (Kurmancî) |      0.9881 |   0.9793 |     0.9837 |       338 |
| knc          | Central Kanuri              |      0.9775 |   1.0000 |     0.9886 |       174 |
| kor          | Korean                      |      1.0000 |   0.9806 |     0.9902 |       360 |
| kzj          | Coastal Kadazan             |      0.9744 |   0.9580 |     0.9661 |       238 |
| lad          | Ladino                      |      0.8154 |   0.8281 |     0.8217 |        64 |
| lat          | Latin                       |      0.9756 |   0.9677 |     0.9717 |       496 |
| lfn          | Lingua Franca Nova          |      0.9745 |   0.9768 |     0.9757 |       431 |
| lij          | Ligurian                    |      0.9556 |   0.9556 |     0.9556 |        90 |
| lin          | Lingala                     |      0.9859 |   0.9859 |     0.9859 |       213 |
| lit          | Lithuanian                  |      0.9903 |   0.9942 |     0.9922 |       513 |
| ltz          | Luxembourgish               |      0.9773 |   0.9149 |     0.9451 |        47 |
| lvs          | Latvian                     |      0.9732 |   0.9797 |     0.9764 |       148 |
| lzh          | Literary Chinese            |      0.7473 |   0.9444 |     0.8344 |        72 |
| mal          | Malayalam                   |      1.0000 |   1.0000 |     1.0000 |        44 |
| mar          | Marathi                     |      0.9961 |   1.0000 |     0.9980 |       509 |
| mhr          | Meadow Mari                 |      0.9899 |   0.9801 |     0.9850 |       201 |
| mkd          | Macedonian                  |      0.9630 |   0.9447 |     0.9538 |       524 |
| mon          | Mongolian                   |      0.9781 |   0.9710 |     0.9745 |       138 |
| mus          | Muskogee (Creek)            |      0.9333 |   0.9655 |     0.9492 |        29 |
| mya          | Burmese                     |      1.0000 |   1.0000 |     1.0000 |        27 |
| nds          | Low German (Low Saxon)      |      0.9829 |   0.9805 |     0.9817 |       410 |
| nld          | Dutch                       |      0.9681 |   0.9810 |     0.9745 |       526 |
| nnb          | Nande                       |      0.9896 |   0.9845 |     0.9870 |       387 |
| nno          | Norwegian Nynorsk           |      0.9551 |   0.9685 |     0.9617 |       571 |
| nob          | Norwegian Bokmål            |      0.9280 |   0.9168 |     0.9224 |       914 |
| nst          | Naga (Tangshang)            |      1.0000 |   1.0000 |     1.0000 |        39 |
| nus          | Nuer                        |      0.9903 |   1.0000 |     0.9951 |       102 |
| oci          | Occitan                     |      0.9795 |   0.9598 |     0.9696 |       249 |
| orv          | Old East Slavic             |      0.9846 |   1.0000 |     0.9922 |        64 |
| oss          | Ossetian                    |      0.9891 |   0.9963 |     0.9927 |       272 |
| ota          | Ottoman Turkish             |      0.9469 |   0.9727 |     0.9596 |       110 |
| pam          | Kapampangan                 |      0.9865 |   0.9733 |     0.9799 |        75 |
| pcd          | Picard                      |      0.9552 |   0.9697 |     0.9624 |        66 |
| pes          | Persian                     |      0.9934 |   0.9956 |     0.9945 |       454 |
| pms          | Piedmontese                 |      0.9268 |   0.9744 |     0.9500 |        39 |
| pol          | Polish                      |      0.9886 |   0.9886 |     0.9886 |       525 |
| por          | Portuguese                  |      0.9669 |   0.9686 |     0.9677 |       542 |
| prg          | Old Prussian                |      0.9800 |   0.9608 |     0.9703 |        51 |
| rhg          | Rohingya                    |      0.9890 |   1.0000 |     0.9945 |       180 |
| rom          | Romani                      |      0.9535 |   0.8913 |     0.9213 |        46 |
| ron          | Romanian                    |      0.9870 |   0.9785 |     0.9827 |       465 |
| run          | Kirundi                     |      0.9871 |   0.9746 |     0.9808 |       236 |
| rus          | Russian                     |      0.9671 |   0.9796 |     0.9733 |       540 |
| sah          | Yakut                       |      1.0000 |   1.0000 |     1.0000 |        48 |
| sat          | Santali                     |      1.0000 |   1.0000 |     1.0000 |       171 |
| sdh          | Southern Kurdish            |      0.9808 |   0.9107 |     0.9444 |        56 |
| shi          | Tashelhit                   |      0.9779 |   0.9172 |     0.9466 |       145 |
| slk          | Slovak                      |      0.9235 |   0.9421 |     0.9327 |       397 |
| slv          | Slovenian                   |      0.7544 |   0.8958 |     0.8190 |        48 |
| sma          | Southern Sami               |      0.9600 |   0.9600 |     0.9600 |       100 |
| sme          | Northern Sami               |      1.0000 |   0.9901 |     0.9950 |       505 |
| smj          | Lule Sami                   |      0.9860 |   1.0000 |     0.9930 |       493 |
| smn          | Inari Sami                  |      0.9950 |   0.9950 |     0.9950 |       200 |
| sms          | Skolt Sami                  |      0.9850 |   0.9899 |     0.9875 |       199 |
| spa          | Spanish                     |      0.9779 |   0.9619 |     0.9698 |       551 |
| sqi          | Albanian                    |      0.9683 |   0.9839 |     0.9760 |       124 |
| srp          | Serbian                     |      0.8347 |   0.8313 |     0.8330 |       492 |
| swc          | Congo Swahili               |      0.8750 |   0.8594 |     0.8671 |       448 |
| swe          | Swedish                     |      0.9809 |   0.9839 |     0.9824 |       991 |
| swg          | Swabian                     |      0.9898 |   0.9604 |     0.9749 |       101 |
| swh          | Swahili                     |      0.6946 |   0.7382 |     0.7157 |       191 |
| tat          | Tatar                       |      0.9817 |   0.9843 |     0.9830 |       382 |
| tgl          | Tagalog                     |      0.9830 |   0.9830 |     0.9830 |       412 |
| tha          | Thai                        |      1.0000 |   1.0000 |     1.0000 |       220 |
| thv          | Tahaggart Tamahaq           |      0.7241 |   0.8400 |     0.7778 |        25 |
| tig          | Tigre                       |      1.0000 |   1.0000 |     1.0000 |       181 |
| tlh          | Klingon                     |      1.0000 |   1.0000 |     1.0000 |       439 |
| tok          | Toki Pona                   |      1.0000 |   1.0000 |     1.0000 |       495 |
| tpw          | Old Tupi                    |      0.8929 |   0.9615 |     0.9259 |        26 |
| tuk          | Turkmen                     |      0.9890 |   0.9711 |     0.9800 |       277 |
| tur          | Turkish                     |      0.9872 |   0.9659 |     0.9764 |       558 |
| uig          | Uyghur                      |      0.9966 |   0.9933 |     0.9950 |       299 |
| ukr          | Ukrainian                   |      0.9813 |   0.9850 |     0.9831 |       532 |
| urd          | Urdu                        |      1.0000 |   0.9914 |     0.9957 |       116 |
| uzb          | Uzbek                       |      0.8200 |   0.9762 |     0.8913 |        42 |
| vie          | Vietnamese                  |      0.9977 |   0.9977 |     0.9977 |       426 |
| vol          | Volapük                     |      0.9862 |   0.9908 |     0.9885 |       217 |
| war          | Waray                       |      0.9505 |   0.9796 |     0.9648 |        98 |
| wuu          | Shanghainese                |      0.8364 |   0.9275 |     0.8796 |       193 |
| xal          | Kalmyk                      |      0.9302 |   0.9756 |     0.9524 |        41 |
| xmf          | Mingrelian                  |      0.7419 |   0.8519 |     0.7931 |        27 |
| yid          | Yiddish                     |      0.9971 |   1.0000 |     0.9986 |       348 |
| yue          | Cantonese                   |      0.9195 |   0.9877 |     0.9524 |       243 |
| zgh          | Standard Moroccan Tamazight |      0.9873 |   0.9873 |     0.9873 |       158 |
| zlm          | Malay (Vernacular)          |      0.8605 |   0.9024 |     0.8810 |        82 |
| zsm          | Malay                       |      0.7782 |   0.7921 |     0.7851 |       279 |
| zza          | Zaza                        |      0.9294 |   0.9294 |     0.9294 |        85 |
| Accuracy     |                             |             |          |     0.9620 |     44049 |
| Weighted avg |                             |      0.9627 |   0.9620 |     0.9621 |     44049 |
| Macro avg    |                             |      0.9434 |   0.9528 |     0.9476 |     44049 |

</details>

### `nb-nordic-lid.ftz`

The small models are quantized versions of the large versions using a cutoff of 50,000 words and ngrams and quantizing the norm separately.

| ISO-639-3    | Language          |   Precision |   Recall |   F1-Score |   Support |
|:-------------|:------------------|------------:|---------:|-----------:|----------:|
| dan          | Danish            |      0.9700 |   0.9838 |     0.9768 |       493 |
| eng          | English           |      0.9980 |   0.9940 |     0.9960 |       502 |
| fao          | Faroese           |      0.9920 |   0.9920 |     0.9920 |       500 |
| fin          | Finnish           |      1.0000 |   1.0000 |     1.0000 |       500 |
| isl          | Icelandic         |      0.9880 |   0.9920 |     0.9900 |       498 |
| nno          | Norwegian Nynorsk |      0.9880 |   0.9841 |     0.9860 |       502 |
| nob          | Norwegian Bokmål  |      0.9860 |   0.9705 |     0.9782 |       508 |
| sma          | Southern Sami     |      0.9800 |   0.9703 |     0.9751 |       101 |
| sme          | Northern Sami     |      1.0000 |   0.9921 |     0.9960 |       504 |
| smj          | Lule Sami         |      0.9920 |   0.9940 |     0.9930 |       499 |
| smn          | Inari Sami        |      0.9950 |   1.0000 |     0.9975 |       199 |
| sms          | Skolt Sami        |      0.9850 |   0.9949 |     0.9899 |       198 |
| swe          | Swedish           |      0.9820 |   0.9899 |     0.9859 |       496 |
| Accuracy     |                   |             |          |     0.9895 |      5500 |
| Weighted avg |                   |      0.9895 |   0.9895 |     0.9895 |      5500 |
| Macro avg    |                   |      0.9889 |   0.9890 |     0.9890 |      5500 |


### `nb-nordic-lid.159.ftz`

<details>
  <summary>Scores for the 159 languages (compressed model)</summary>

| ISO-639-3    | Language                    |   Precision |   Recall |   F1-Score |   Support |
|:-------------|:----------------------------|------------:|---------:|-----------:|----------:|
| afr          | Afrikaans                   |      0.9529 |   0.9333 |     0.9430 |       195 |
| ara          | Arabic                      |      0.9708 |   0.9191 |     0.9443 |       507 |
| arq          | Algerian Arabic             |      0.8783 |   0.8783 |     0.8783 |       115 |
| arz          | Egyptian Arabic             |      0.5439 |   0.8378 |     0.6596 |        37 |
| asm          | Assamese                    |      0.9828 |   0.9448 |     0.9634 |       181 |
| avk          | Kotava                      |      0.9843 |   0.9792 |     0.9817 |       192 |
| aze          | Azerbaijani                 |      0.9582 |   0.9828 |     0.9703 |       233 |
| bel          | Belarusian                  |      0.9919 |   0.9683 |     0.9799 |       378 |
| ben          | Bengali                     |      0.9574 |   0.9868 |     0.9719 |       228 |
| ber          | Berber                      |      0.8495 |   0.7928 |     0.8202 |       584 |
| bos          | Bosnian                     |      0.1429 |   0.2264 |     0.1752 |        53 |
| bre          | Breton                      |      0.9507 |   0.9712 |     0.9609 |       278 |
| bua          | Buryat                      |      0.9333 |   0.9333 |     0.9333 |        45 |
| bul          | Bulgarian                   |      0.9351 |   0.9457 |     0.9404 |       442 |
| cat          | Catalan                     |      0.9406 |   0.9406 |     0.9406 |       303 |
| cbk          | Chavacano                   |      0.9552 |   0.9624 |     0.9588 |       133 |
| ceb          | Cebuano                     |      0.8718 |   0.8500 |     0.8608 |        80 |
| ces          | Czech                       |      0.9586 |   0.9548 |     0.9567 |       509 |
| chv          | Chuvash                     |      1.0000 |   0.9647 |     0.9820 |        85 |
| ckb          | Central Kurdish (Soranî)    |      0.9640 |   0.9748 |     0.9694 |       357 |
| ckt          | Chukchi                     |      0.9615 |   1.0000 |     0.9804 |        25 |
| cmn          | Mandarin Chinese            |      0.9667 |   0.8165 |     0.8853 |       605 |
| cor          | Cornish                     |      0.9780 |   0.9674 |     0.9727 |       184 |
| csb          | Kashubian                   |      0.9574 |   1.0000 |     0.9783 |        45 |
| cym          | Welsh                       |      0.9625 |   0.9506 |     0.9565 |        81 |
| dan          | Danish                      |      0.9281 |   0.9355 |     0.9318 |       993 |
| deu          | German                      |      0.9853 |   0.9781 |     0.9817 |       549 |
| dsb          | Lower Sorbian               |      0.8889 |   0.8276 |     0.8571 |        58 |
| dtp          | Central Dusun               |      0.8741 |   0.9470 |     0.9091 |       132 |
| ell          | Greek                       |      0.9958 |   0.9937 |     0.9947 |       476 |
| eng          | English                     |      0.9886 |   0.9876 |     0.9881 |      1050 |
| epo          | Esperanto                   |      0.9853 |   0.9818 |     0.9835 |       548 |
| est          | Estonian                    |      0.9489 |   0.9766 |     0.9625 |       171 |
| eus          | Basque                      |      0.9844 |   0.9583 |     0.9712 |       264 |
| fao          | Faroese                     |      0.9780 |   0.9819 |     0.9800 |       498 |
| fin          | Finnish                     |      0.9922 |   0.9724 |     0.9822 |      1050 |
| fkv          | Kven Finnish                |      0.5385 |   0.7368 |     0.6222 |        19 |
| fra          | French                      |      0.9871 |   0.9728 |     0.9799 |       552 |
| frr          | North Frisian               |      0.9640 |   0.9640 |     0.9640 |       139 |
| fry          | Frisian                     |      0.7097 |   0.8462 |     0.7719 |        26 |
| gcf          | Guadeloupean Creole French  |      0.9714 |   0.9808 |     0.9761 |       104 |
| gla          | Scottish Gaelic             |      0.9608 |   0.9608 |     0.9608 |        51 |
| gle          | Irish                       |      0.9489 |   0.9924 |     0.9701 |       131 |
| glg          | Galician                    |      0.8868 |   0.9082 |     0.8974 |       207 |
| gos          | Gronings                    |      0.9426 |   0.9544 |     0.9485 |       241 |
| grc          | Ancient Greek               |      0.9483 |   0.9483 |     0.9483 |        58 |
| grn          | Guarani                     |      0.9684 |   0.9935 |     0.9808 |       154 |
| guc          | Wayuu                       |      0.9333 |   1.0000 |     0.9655 |        42 |
| hau          | Hausa                       |      0.9861 |   0.9884 |     0.9872 |       430 |
| heb          | Hebrew                      |      0.9981 |   0.9907 |     0.9944 |       540 |
| hin          | Hindi                       |      0.9974 |   0.9898 |     0.9936 |       393 |
| hoc          | Ho                          |      0.8571 |   1.0000 |     0.9231 |        30 |
| hrv          | Croatian                    |      0.6766 |   0.5911 |     0.6310 |       269 |
| hrx          | Hunsrik                     |      0.8545 |   0.9216 |     0.8868 |        51 |
| hsb          | Upper Sorbian               |      0.8400 |   0.8182 |     0.8289 |        77 |
| hun          | Hungarian                   |      0.9816 |   0.9852 |     0.9834 |       541 |
| hye          | Armenian                    |      1.0000 |   1.0000 |     1.0000 |       225 |
| ido          | Ido                         |      0.9672 |   0.9501 |     0.9586 |       341 |
| ile          | Interlingue                 |      0.9352 |   0.9547 |     0.9448 |       287 |
| ilo          | Ilocano                     |      0.9917 |   0.9600 |     0.9756 |       125 |
| ina          | Interlingua                 |      0.9580 |   0.9558 |     0.9569 |       453 |
| ind          | Indonesian                  |      0.8231 |   0.8034 |     0.8131 |       417 |
| isl          | Icelandic                   |      0.9805 |   0.9885 |     0.9845 |       867 |
| ita          | Italian                     |      0.9817 |   0.9555 |     0.9684 |       562 |
| jav          | Javanese                    |      0.9400 |   0.9792 |     0.9592 |        48 |
| jbo          | Lojban                      |      1.0000 |   0.9975 |     0.9988 |       403 |
| jpn          | Japanese                    |      0.9684 |   0.9981 |     0.9830 |       522 |
| kab          | Kabyle                      |      0.7702 |   0.8516 |     0.8089 |       492 |
| kat          | Georgian                    |      1.0000 |   0.9847 |     0.9923 |       261 |
| kaz          | Kazakh                      |      0.9792 |   0.9843 |     0.9817 |       191 |
| kha          | Khasi                       |      0.8942 |   0.9300 |     0.9118 |       100 |
| khm          | Khmer                       |      1.0000 |   0.9868 |     0.9934 |        76 |
| kmr          | Northern Kurdish (Kurmancî) |      0.9791 |   0.9647 |     0.9719 |       340 |
| knc          | Central Kanuri              |      0.9775 |   0.9943 |     0.9858 |       175 |
| kor          | Korean                      |      0.9972 |   0.9778 |     0.9874 |       360 |
| kzj          | Coastal Kadazan             |      0.9658 |   0.9378 |     0.9516 |       241 |
| lad          | Ladino                      |      0.7538 |   0.8033 |     0.7778 |        61 |
| lat          | Latin                       |      0.9614 |   0.9594 |     0.9604 |       493 |
| lfn          | Lingua Franca Nova          |      0.9722 |   0.9611 |     0.9666 |       437 |
| lij          | Ligurian                    |      0.8778 |   0.9753 |     0.9240 |        81 |
| lin          | Lingala                     |      0.9859 |   0.9677 |     0.9767 |       217 |
| lit          | Lithuanian                  |      0.9864 |   0.9864 |     0.9864 |       515 |
| ltz          | Luxembourgish               |      0.9773 |   0.9149 |     0.9451 |        47 |
| lvs          | Latvian                     |      0.9597 |   0.9662 |     0.9630 |       148 |
| lzh          | Literary Chinese            |      0.6593 |   0.8108 |     0.7273 |        74 |
| mal          | Malayalam                   |      1.0000 |   1.0000 |     1.0000 |        44 |
| mar          | Marathi                     |      0.9902 |   0.9980 |     0.9941 |       507 |
| mhr          | Meadow Mari                 |      0.9899 |   0.9752 |     0.9825 |       202 |
| mkd          | Macedonian                  |      0.9397 |   0.9253 |     0.9324 |       522 |
| mon          | Mongolian                   |      0.9781 |   0.9571 |     0.9675 |       140 |
| mus          | Muskogee (Creek)            |      0.9000 |   0.9643 |     0.9310 |        28 |
| mya          | Burmese                     |      1.0000 |   1.0000 |     1.0000 |        27 |
| nds          | Low German (Low Saxon)      |      0.9829 |   0.9687 |     0.9757 |       415 |
| nld          | Dutch                       |      0.9644 |   0.9735 |     0.9689 |       528 |
| nnb          | Nande                       |      0.9870 |   0.9896 |     0.9883 |       384 |
| nno          | Norwegian Nynorsk           |      0.9499 |   0.9632 |     0.9565 |       571 |
| nob          | Norwegian Bokmål            |      0.9324 |   0.9073 |     0.9197 |       928 |
| nst          | Naga (Tangshang)            |      1.0000 |   0.9750 |     0.9873 |        40 |
| nus          | Nuer                        |      0.9903 |   1.0000 |     0.9951 |       102 |
| oci          | Occitan                     |      0.9631 |   0.9476 |     0.9553 |       248 |
| orv          | Old East Slavic             |      0.9538 |   0.9254 |     0.9394 |        67 |
| oss          | Ossetian                    |      0.9818 |   0.9926 |     0.9872 |       271 |
| ota          | Ottoman Turkish             |      0.9204 |   0.9455 |     0.9327 |       110 |
| pam          | Kapampangan                 |      0.9730 |   0.9600 |     0.9664 |        75 |
| pcd          | Picard                      |      0.9254 |   0.9688 |     0.9466 |        64 |
| pes          | Persian                     |      0.9846 |   0.9868 |     0.9857 |       454 |
| pms          | Piedmontese                 |      0.9024 |   0.9487 |     0.9250 |        39 |
| pol          | Polish                      |      0.9867 |   0.9885 |     0.9876 |       524 |
| por          | Portuguese                  |      0.9595 |   0.9577 |     0.9586 |       544 |
| prg          | Old Prussian                |      0.9800 |   0.9423 |     0.9608 |        52 |
| rhg          | Rohingya                    |      0.9835 |   0.9835 |     0.9835 |       182 |
| rom          | Romani                      |      0.9302 |   0.8511 |     0.8889 |        47 |
| ron          | Romanian                    |      0.9783 |   0.9762 |     0.9772 |       462 |
| run          | Kirundi                     |      0.9871 |   0.9426 |     0.9644 |       244 |
| rus          | Russian                     |      0.9561 |   0.9757 |     0.9658 |       536 |
| sah          | Yakut                       |      0.9792 |   1.0000 |     0.9895 |        47 |
| sat          | Santali                     |      0.9942 |   1.0000 |     0.9971 |       170 |
| sdh          | Southern Kurdish            |      0.8462 |   0.8627 |     0.8544 |        51 |
| shi          | Tashelhit                   |      0.9706 |   0.8980 |     0.9329 |       147 |
| slk          | Slovak                      |      0.9111 |   0.9318 |     0.9213 |       396 |
| slv          | Slovenian                   |      0.7018 |   0.9302 |     0.8000 |        43 |
| sma          | Southern Sami               |      0.9500 |   0.9406 |     0.9453 |       101 |
| sme          | Northern Sami               |      1.0000 |   0.9843 |     0.9921 |       508 |
| smj          | Lule Sami                   |      0.9840 |   0.9980 |     0.9909 |       493 |
| smn          | Inari Sami                  |      0.9850 |   0.9949 |     0.9899 |       198 |
| sms          | Skolt Sami                  |      0.9700 |   0.9848 |     0.9773 |       197 |
| spa          | Spanish                     |      0.9613 |   0.9560 |     0.9586 |       545 |
| sqi          | Albanian                    |      0.9603 |   0.9680 |     0.9641 |       125 |
| srp          | Serbian                     |      0.8122 |   0.8106 |     0.8114 |       491 |
| swc          | Congo Swahili               |      0.8500 |   0.8367 |     0.8433 |       447 |
| swe          | Swedish                     |      0.9759 |   0.9778 |     0.9768 |       992 |
| swg          | Swabian                     |      0.9796 |   0.9320 |     0.9552 |       103 |
| swh          | Swahili                     |      0.6650 |   0.7068 |     0.6853 |       191 |
| tat          | Tatar                       |      0.9739 |   0.9816 |     0.9777 |       380 |
| tgl          | Tagalog                     |      0.9709 |   0.9732 |     0.9721 |       411 |
| tha          | Thai                        |      1.0000 |   1.0000 |     1.0000 |       220 |
| thv          | Tahaggart Tamahaq           |      0.6552 |   0.7600 |     0.7037 |        25 |
| tig          | Tigre                       |      1.0000 |   1.0000 |     1.0000 |       181 |
| tlh          | Klingon                     |      0.9977 |   0.9955 |     0.9966 |       440 |
| tok          | Toki Pona                   |      1.0000 |   1.0000 |     1.0000 |       495 |
| tpw          | Old Tupi                    |      0.8214 |   0.8846 |     0.8519 |        26 |
| tuk          | Turkmen                     |      0.9779 |   0.9708 |     0.9744 |       274 |
| tur          | Turkish                     |      0.9780 |   0.9604 |     0.9691 |       556 |
| uig          | Uyghur                      |      0.9933 |   0.9900 |     0.9916 |       299 |
| ukr          | Ukrainian                   |      0.9682 |   0.9700 |     0.9691 |       533 |
| urd          | Urdu                        |      1.0000 |   0.9914 |     0.9957 |       116 |
| uzb          | Uzbek                       |      0.8000 |   0.9756 |     0.8791 |        41 |
| vie          | Vietnamese                  |      0.9977 |   0.9977 |     0.9977 |       426 |
| vol          | Volapük                     |      0.9862 |   0.9817 |     0.9840 |       219 |
| war          | Waray                       |      0.9208 |   0.9688 |     0.9442 |        96 |
| wuu          | Shanghainese                |      0.8037 |   0.9053 |     0.8515 |       190 |
| xal          | Kalmyk                      |      0.9070 |   0.9512 |     0.9286 |        41 |
| xmf          | Mingrelian                  |      0.6774 |   0.8400 |     0.7500 |        25 |
| yid          | Yiddish                     |      0.9828 |   0.9942 |     0.9885 |       345 |
| yue          | Cantonese                   |      0.8314 |   0.9688 |     0.8948 |       224 |
| zgh          | Standard Moroccan Tamazight |      0.9873 |   0.9873 |     0.9873 |       158 |
| zlm          | Malay (Vernacular)          |      0.8488 |   0.8588 |     0.8538 |        85 |
| zsm          | Malay                       |      0.7465 |   0.7544 |     0.7504 |       281 |
| zza          | Zaza                        |      0.8824 |   0.9146 |     0.8982 |        82 |
| Accuracy     |                             |             |          |     0.9513 |     44049 |
| Weighted avg |                             |      0.9529 |   0.9513 |     0.9518 |     44049 |
| Macro avg    |                             |      0.9275 |   0.9399 |     0.9327 |     44049 |

</details>


## Citing & Authors

The model was trained by Javier de la Rosa. Data was prepared by Per Egil Kummervold and Javier de la Rosa. Documentation written by Javier de la Rosa.