README.md · wikilangs/azb at main

azb / README.md

omarkamali

Upload all models and assets for azb (latest)

42db662 verified about 2 months ago

preview code

raw

history blame contribute delete

32.7 kB

	---
	language: azb
	language_name: South Azerbaijani
	language_family: turkic_oghuz
	tags:
	- wikilangs
	- nlp
	- tokenizer
	- embeddings
	- n-gram
	- markov
	- wikipedia
	- feature-extraction
	- sentence-similarity
	- tokenization
	- n-grams
	- markov-chain
	- text-mining
	- fasttext
	- babelvec
	- vocabulous
	- vocabulary
	- monolingual
	- family-turkic_oghuz
	license: mit
	library_name: wikilangs
	pipeline_tag: text-generation
	datasets:
	- omarkamali/wikipedia-monthly
	dataset_info:
	name: wikipedia-monthly
	description: Monthly snapshots of Wikipedia articles across 300+ languages
	metrics:
	- name: best_compression_ratio
	type: compression
	value: 4.154
	- name: best_isotropy
	type: isotropy
	value: 0.8266
	- name: vocabulary_size
	type: vocab
	value: 0
	generated: 2026-01-03
	---

	# South Azerbaijani - Wikilangs Models
	## Comprehensive Research Report & Full Ablation Study

	This repository contains NLP models trained and evaluated by Wikilangs, specifically on South Azerbaijani Wikipedia data.
	We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and word embeddings.

	## 📋 Repository Contents

	### Models & Assets

	- Tokenizers (8k, 16k, 32k, 64k)
	- N-gram models (2, 3, 4, 5-gram)
	- Markov chains (context of 1, 2, 3, 4 and 5)
	- Subword N-gram and Markov chains
	- Embeddings in various sizes and dimensions (aligned and unaligned)
	- Language Vocabulary
	- Language Statistics

	![Performance Dashboard](visualizations/performance_dashboard.png)

	### Analysis and Evaluation

	- [1. Tokenizer Evaluation](#1-tokenizer-evaluation)
	- [2. N-gram Model Evaluation](#2-n-gram-model-evaluation)
	- [3. Markov Chain Evaluation](#3-markov-chain-evaluation)
	- [4. Vocabulary Analysis](#4-vocabulary-analysis)
	- [5. Word Embeddings Evaluation](#5-word-embeddings-evaluation)
	- [6. Morphological Analysis (Experimental)](#6--morphological-analysis-experimental)
	- [7. Summary & Recommendations](#7-summary--recommendations)
	- [Metrics Glossary](#appendix-metrics-glossary--interpretation-guide)
	- [Visualizations Index](#visualizations-index)

	---
	## 1. Tokenizer Evaluation

	![Tokenizer Compression](visualizations/tokenizer_compression.png)

	![Tokenizer Fertility](visualizations/tokenizer_fertility.png)

	![Tokenizer OOV](visualizations/tokenizer_oov.png)

	![Total Tokens](visualizations/tokenizer_total_tokens.png)

	### Results

	\| Vocab Size \| Compression \| Avg Token Len \| UNK Rate \| Total Tokens \|
	\|------------\|-------------\|---------------\|----------\|--------------\|
	\| 8k \| 3.140x \| 3.14 \| 0.4916% \| 361,073 \|
	\| 16k \| 3.514x \| 3.52 \| 0.5501% \| 322,683 \|
	\| 32k \| 3.859x \| 3.86 \| 0.6041% \| 293,823 \|
	\| 64k \| 4.154x 🏆 \| 4.16 \| 0.6502% \| 272,975 \|

	### Tokenization Examples

	Below are sample sentences tokenized with each vocabulary size:

	Sample 1: `پوچتوووی ( ) روسیه اؤلکه‌سینده یئر آلان بیر کند دیر و آرخانقلسک اوبلاستیندا یئرل...`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁پو چ ت وو وی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ... (+12 more)` \| 22 \|
	\| 16k \| `▁پو چ ت وو وی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ... (+12 more)` \| 22 \|
	\| 32k \| `▁پو چت وووی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ▁آلان ... (+10 more)` \| 20 \|
	\| 64k \| `▁پو چت وووی ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ▁آلان ... (+10 more)` \| 20 \|

	Sample 2: `هیندوستان اؤلکه‌سینین کارناتاکا ایالتینده بیر کند دیر. بۇ کنده کانادا دیلی دانیش...`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁هیندوستان ▁اؤلکه ▁سینین ▁کارناتاکا ▁ایالتینده ▁بیر ▁کند ▁دیر . ▁بۇ ... (+7 more)` \| 17 \|
	\| 16k \| `▁هیندوستان ▁اؤلکه ▁سینین ▁کارناتاکا ▁ایالتینده ▁بیر ▁کند ▁دیر . ▁بۇ ... (+7 more)` \| 17 \|
	\| 32k \| `▁هیندوستان ▁اؤلکه ▁سینین ▁کارناتاکا ▁ایالتینده ▁بیر ▁کند ▁دیر . ▁بۇ ... (+7 more)` \| 17 \|
	\| 64k \| `▁هیندوستان ▁اؤلکه ▁سینین ▁کارناتاکا ▁ایالتینده ▁بیر ▁کند ▁دیر . ▁بۇ ... (+7 more)` \| 17 \|

	Sample 3: `پیایو، روسیه ( ) روسیه اؤلکه‌سینده یئر آلان بیر کند دیر و مورمانسک اوبلاستیندا ی...`

	\| Vocab \| Tokens \| Count \|
	\|-------\|--------\|-------\|
	\| 8k \| `▁پی ای و ، ▁روسیه ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ... (+14 more)` \| 24 \|
	\| 16k \| `▁پی ای و ، ▁روسیه ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ... (+12 more)` \| 22 \|
	\| 32k \| `▁پی ایو ، ▁روسیه ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ... (+11 more)` \| 21 \|
	\| 64k \| `▁پی ایو ، ▁روسیه ▁( ▁) ▁روسیه ▁اؤلکه ▁سینده ▁یئر ... (+11 more)` \| 21 \|


	### Key Findings

	- Best Compression: 64k achieves 4.154x compression
	- Lowest UNK Rate: 8k with 0.4916% unknown tokens
	- Trade-off: Larger vocabularies improve compression but increase model size
	- Recommendation: 32k vocabulary provides optimal balance for production use

	---
	## 2. N-gram Model Evaluation

	![N-gram Perplexity](visualizations/ngram_perplexity.png)

	![N-gram Unique](visualizations/ngram_unique.png)

	![N-gram Coverage](visualizations/ngram_coverage.png)

	### Results

	\| N-gram \| Variant \| Perplexity \| Entropy \| Unique N-grams \| Top-100 Coverage \| Top-1000 Coverage \|
	\|--------\|---------\|------------\|---------\|----------------\|------------------\|-------------------\|
	\| 2-gram \| Word \| 8,053 \| 12.98 \| 158,968 \| 25.7% \| 56.1% \|
	\| 2-gram \| Subword \| 528 🏆 \| 9.04 \| 12,667 \| 51.7% \| 95.7% \|
	\| 3-gram \| Word \| 10,252 \| 13.32 \| 236,817 \| 22.6% \| 53.6% \|
	\| 3-gram \| Subword \| 3,765 \| 11.88 \| 106,797 \| 23.2% \| 62.4% \|
	\| 4-gram \| Word \| 17,203 \| 14.07 \| 427,241 \| 19.0% \| 47.9% \|
	\| 4-gram \| Subword \| 15,109 \| 13.88 \| 582,040 \| 14.6% \| 44.8% \|
	\| 5-gram \| Word \| 19,665 \| 14.26 \| 390,296 \| 17.2% \| 45.0% \|
	\| 5-gram \| Subword \| 37,921 \| 15.21 \| 1,605,166 \| 11.7% \| 37.8% \|

	### Top 5 N-grams by Size

	2-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `ایشلدنلری طرفیندن` \| 75,586 \|
	\| 2 \| `مقاله‌سیندن گؤتورولوبدور` \| 75,505 \|
	\| 3 \| `ویکی‌پدیاسی‌نین ایشلدنلری` \| 73,736 \|
	\| 4 \| `اینگیلیسجه ویکی‌پدیاسی‌نین` \| 71,134 \|
	\| 5 \| `قایناق‌لار اینگیلیسجه` \| 70,887 \|

	3-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن` \| 73,736 \|
	\| 2 \| `اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری` \| 71,134 \|
	\| 3 \| `قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین` \| 70,813 \|
	\| 4 \| `بیر یاشاییش منطقه‌سی‌دیر` \| 40,398 \|
	\| 5 \| `بیر کند دیر` \| 30,448 \|

	4-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن` \| 71,134 \|
	\| 2 \| `قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری` \| 70,813 \|
	\| 3 \| `سوْن نۆفوس ساییمی اساسيندا` \| 24,567 \|
	\| 4 \| `شهرلرین لیستی قایناق‌لار اینگیلیسجه` \| 22,936 \|
	\| 5 \| `لیستی قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین` \| 22,936 \|

	5-grams (Word):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن` \| 70,813 \|
	\| 2 \| `شهرلرین لیستی قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین` \| 22,936 \|
	\| 3 \| `لیستی قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری` \| 22,936 \|
	\| 4 \| `گؤتورولوبدور ۸ آقوست تاریخینده یوْخلانیلیبدیر` \| 17,804 \|
	\| 5 \| `مقاله‌سیندن گؤتورولوبدور ۸ آقوست تاریخینده` \| 17,804 \|

	2-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `ی ن` \| 1,870,834 \|
	\| 2 \| `_ ا` \| 1,661,040 \|
	\| 3 \| `ی _` \| 1,438,867 \|
	\| 4 \| `ا ی` \| 1,394,440 \|
	\| 5 \| `ن _` \| 1,218,001 \|

	3-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `_ ا ی` \| 718,299 \|
	\| 2 \| `ی ن د` \| 659,203 \|
	\| 3 \| `د ه _` \| 586,025 \|
	\| 4 \| `ل ا ر` \| 581,126 \|
	\| 5 \| `ا ی ن` \| 470,804 \|

	4-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `ن د ه _` \| 347,491 \|
	\| 2 \| `ل ا ر _` \| 329,654 \|
	\| 3 \| `ی ن د ه` \| 321,093 \|
	\| 4 \| `_ ب ی ر` \| 258,934 \|
	\| 5 \| `ن ی ن _` \| 257,847 \|

	5-grams (Subword):

	\| Rank \| N-gram \| Count \|
	\|------\|--------\|-------\|
	\| 1 \| `ی ن د ه _` \| 319,618 \|
	\| 2 \| `ق ا ی ن ا` \| 236,232 \|
	\| 3 \| `_ ق ا ی ن` \| 235,936 \|
	\| 4 \| `ی ن د ن _` \| 199,409 \|
	\| 5 \| `ی ن گ ی ل` \| 172,619 \|


	### Key Findings

	- Best Perplexity: 2-gram (subword) with 528
	- Entropy Trend: Decreases with larger n-grams (more predictable)
	- Coverage: Top-1000 patterns cover ~38% of corpus
	- Recommendation: 4-gram or 5-gram for best predictive performance

	---
	## 3. Markov Chain Evaluation

	![Markov Entropy](visualizations/markov_entropy.png)

	![Markov Contexts](visualizations/markov_contexts.png)

	![Markov Branching](visualizations/markov_branching.png)

	### Results

	\| Context \| Variant \| Avg Entropy \| Perplexity \| Branching Factor \| Unique Contexts \| Predictability \|
	\|---------\|---------\|-------------\|------------\|------------------\|-----------------\|----------------\|
	\| 1 \| Word \| 0.6633 \| 1.584 \| 5.08 \| 728,851 \| 33.7% \|
	\| 1 \| Subword \| 1.0599 \| 2.085 \| 9.06 \| 3,419 \| 0.0% \|
	\| 2 \| Word \| 0.1969 \| 1.146 \| 1.48 \| 3,698,118 \| 80.3% \|
	\| 2 \| Subword \| 0.9299 \| 1.905 \| 6.57 \| 30,973 \| 7.0% \|
	\| 3 \| Word \| 0.0689 \| 1.049 \| 1.14 \| 5,453,316 \| 93.1% \|
	\| 3 \| Subword \| 0.8407 \| 1.791 \| 4.70 \| 203,342 \| 15.9% \|
	\| 4 \| Word \| 0.0340 🏆 \| 1.024 \| 1.07 \| 6,184,673 \| 96.6% \|
	\| 4 \| Subword \| 0.6999 \| 1.624 \| 3.22 \| 955,119 \| 30.0% \|

	### Generated Text Samples (Word-based)

	Below are text samples generated from each word-based Markov chain model:

	Context Size 1:

	1. `و اؤلوملر باخیر دؤولت موزیک دوْغوملار ۲ مئی eamon gilmore shooter young artist awardbest breakthroug...`
	2. `بیر فوتبالیست هوجومچو موقعیتینده اوْیناییب قایناقلار ایلده آمریکالی سیاستچیلر میلادی ایلده آذربایجان...`
	3. `اینگیلیسجه phyllis and the new york آمریکانین نبراسکا ایالتینده بیر شهردیر و باتی آجورلو قصبه‌سینده ...`

	Context Size 2:

	1. `ایشلدنلری طرفیندن ballard مقاله‌سیندن گؤتورولوبدور ۳۰ نوْوامبر تاریخینده رالی قوزئی کارولینا ایالتین...`
	2. `مقاله‌سیندن گؤتورولوبدور ۲۲ آقوست تاریخینده یوْخلانیلیبدیر ایالتین شهرلری آمریکا بیرلشمیش ایالتلری ک...`
	3. `ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن piguet مقاله‌سیندن گؤتورولوبدور ۱۹ جولای یوْخلانیلیبدیر شهرلری en ...`

	Context Size 3:

	1. `ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن phosphate مقاله‌سیندن گؤتورولوبدور ۳۰ نوْوامبر تاریخینده یوْخلانیل...`
	2. `اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن mała مقاله‌سیندن گؤتورولوبدور ۱۲ آقوست تاریخینده یوْخلا...`
	3. `قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن nigra مقاله‌سیندن گؤتورولوبدور ۲۷ جولای تاری...`

	Context Size 4:

	1. `اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن tachov district مقاله‌سیندن گؤتورولوبدور ۱۹ جولای یوْخل...`
	2. `قایناق‌لار اینگیلیسجه ویکی‌پدیاسی‌نین ایشلدنلری طرفیندن reed مقاله‌سیندن گؤتورولوبدور ۲۲ ژانویه تاری...`
	3. `سوْن نۆفوس ساییمی اساسيندا نفر ایمیش قایناقلار جومهوریتینین شهرلری en bədəlan`


	### Generated Text Samples (Subword-based)

	Below are text samples generated from each subword-based Markov chain model:

	Context Size 1:

	1. `_طین_اینالرشه_می`
	2. `ینینویره_ب.st_آذ`
	3. `اؤلیلده_s_مول_کل`

	Context Size 2:

	1. `ینی_اوبونیرلرین_ش`
	2. `_این_چاری_اوربّع_د`
	3. `ی_حؤکواءنینه‌سیناق`

	Context Size 3:

	1. `_ایشتیرامبر_charah`
	2. `ینده_یئرلشیرکت_()_`
	3. `ده_یوْخلو_"_the_ism`

	Context Size 4:

	1. `نده_یئرلشیر._بۇ_شهر`
	2. `لار_اینسانی._۲۴_آقو`
	3. `ینده_چیخماق_شکیلات)`


	### Key Findings

	- Best Predictability: Context-4 (word) with 96.6% predictability
	- Branching Factor: Decreases with context size (more deterministic)
	- Memory Trade-off: Larger contexts require more storage (955,119 contexts)
	- Recommendation: Context-3 or Context-4 for text generation

	---
	## 4. Vocabulary Analysis

	![Zipf's Law](visualizations/zipf_law.png)

	![Top Words](visualizations/top20_words.png)

	![Coverage Curve](visualizations/vocab_coverage.png)

	### Statistics

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Vocabulary Size \| 271,726 \|
	\| Total Tokens \| 12,485,100 \|
	\| Mean Frequency \| 45.95 \|
	\| Median Frequency \| 3 \|
	\| Frequency Std Dev \| 1144.86 \|

	### Most Common Words

	\| Rank \| Word \| Frequency \|
	\|------\|------\|-----------\|
	\| 1 \| و \| 284,866 \|
	\| 2 \| بیر \| 169,436 \|
	\| 3 \| اینگیلیسجه \| 149,744 \|
	\| 4 \| قایناقلار \| 142,037 \|
	\| 5 \| the \| 114,223 \|
	\| 6 \| تاریخینده \| 92,091 \|
	\| 7 \| قایناق‌لار \| 90,964 \|
	\| 8 \| ایلده \| 83,776 \|
	\| 9 \| شهرلری \| 81,908 \|
	\| 10 \| طرفیندن \| 80,193 \|

	### Least Common Words (from vocabulary)

	\| Rank \| Word \| Frequency \|
	\|------\|------\|-----------\|
	\| 1 \| ائششکین \| 2 \|
	\| 2 \| لابی‌سی \| 2 \|
	\| 3 \| آذری‌ها \| 2 \|
	\| 4 \| داشناکلارلا \| 2 \|
	\| 5 \| قۇلان \| 2 \|
	\| 6 \| آسینۇس \| 2 \|
	\| 7 \| ائششه‌یینین \| 2 \|
	\| 8 \| تاپؽلمیشدیر \| 2 \|
	\| 9 \| kulan \| 2 \|
	\| 10 \| کسا \| 2 \|

	### Zipf's Law Analysis

	\| Metric \| Value \|
	\|--------\|-------\|
	\| Zipf Coefficient \| 1.1608 \|
	\| R² (Goodness of Fit) \| 0.995522 \|
	\| Adherence Quality \| excellent \|

	### Coverage Analysis

	\| Top N Words \| Coverage \|
	\|-------------\|----------\|
	\| Top 100 \| 34.4% \|
	\| Top 1,000 \| 64.8% \|
	\| Top 5,000 \| 79.6% \|
	\| Top 10,000 \| 84.6% \|

	### Key Findings

	- Zipf Compliance: R²=0.9955 indicates excellent adherence to Zipf's law
	- High Frequency Dominance: Top 100 words cover 34.4% of corpus
	- Long Tail: 261,726 words needed for remaining 15.4% coverage

	---
	## 5. Word Embeddings Evaluation

	![Embedding Isotropy](visualizations/embedding_isotropy.png)

	![Similarity Matrix](visualizations/embedding_similarity.png)

	![t-SNE Words](visualizations/tsne_words.png)

	![t-SNE Sentences](visualizations/tsne_sentences.png)


	### 5.1 Cross-Lingual Alignment

	![Alignment Quality](visualizations/embedding_alignment_quality.png)

	![Multilingual t-SNE](visualizations/embedding_tsne_multilingual.png)


	### 5.2 Model Comparison

	\| Model \| Dimension \| Isotropy \| Semantic Density \| Alignment R@1 \| Alignment R@10 \|
	\|-------\|-----------\|----------\|------------------\|---------------\|----------------\|
	\| mono_32d \| 32 \| 0.8266 🏆 \| 0.3562 \| N/A \| N/A \|
	\| mono_64d \| 64 \| 0.7978 \| 0.2932 \| N/A \| N/A \|
	\| mono_128d \| 128 \| 0.7560 \| 0.2495 \| N/A \| N/A \|
	\| aligned_32d \| 32 \| 0.8266 \| 0.3594 \| 0.0580 \| 0.2760 \|
	\| aligned_64d \| 64 \| 0.7978 \| 0.3041 \| 0.1220 \| 0.4360 \|
	\| aligned_128d \| 128 \| 0.7560 \| 0.2442 \| 0.2380 \| 0.6200 \|

	### Key Findings

	- Best Isotropy: mono_32d with 0.8266 (more uniform distribution)
	- Semantic Density: Average pairwise similarity of 0.3011. Lower values indicate better semantic separation.
	- Alignment Quality: Aligned models achieve up to 23.8% R@1 in cross-lingual retrieval.
	- Recommendation: 128d aligned for best cross-lingual performance

	---
	## 6. Morphological Analysis (Experimental)

	This section presents an automated morphological analysis derived from the statistical divergence between word-level and subword-level models. By analyzing where subword predictability spikes and where word-level coverage fails, we can infer linguistic structures without supervised data.

	### 6.1 Productivity & Complexity

	\| Metric \| Value \| Interpretation \| Recommendation \|
	\|--------\|-------\|----------------\|----------------\|
	\| Productivity Index \| 5.000 \| High morphological productivity \| Reliable analysis \|
	\| Idiomaticity Gap \| 0.190 \| Low formulaic content \| - \|

	### 6.2 Affix Inventory (Productive Units)

	These are the most productive prefixes and suffixes identified by sampling the vocabulary for global substitutability patterns. A unit is considered an affix if stripping it leaves a valid stem that appears in other contexts.

	#### Productive Prefixes
	\| Prefix \| Examples \|
	\|--------\|----------\|

	#### Productive Suffixes
	\| Suffix \| Examples \|
	\|--------\|----------\|
	\| `-ین` \| کومیته‌سی‌‌نین, قورانین, پاقلئنین \|
	\| `-ان` \| قافقازدان, آتاسین‌دان, تیتانلاردان \|

	### 6.3 Bound Stems (Lexical Roots)

	Bound stems are high-frequency subword units that are semantically cohesive but rarely appear as standalone words. These often correspond to the 'core' of a word that requires inflection or derivation to be valid.

	\| Stem \| Cohesion \| Substitutability \| Examples \|
	\|------\|----------\|------------------\|----------\|
	\| `رلری` \| 2.05x \| 205 contexts \| یرلری, ارلری, خطرلری \|
	\| `اقلا` \| 1.90x \| 131 contexts \| ناقلا, اقلایس, داقلاس \|
	\| `یبدی` \| 2.33x \| 31 contexts \| ییبدیر, گلیبدی, آلیبدی \|
	\| `قلار` \| 2.01x \| 54 contexts \| حقلاری, لیقلار, حاقلار \|
	\| `اریخ` \| 2.11x \| 41 contexts \| تاریخ, ‌تاریخ, تواریخ \|
	\| `ولوب` \| 1.85x \| 60 contexts \| اولوب, قولوب, بولوب \|
	\| `تیند` \| 1.73x \| 73 contexts \| تینده, تیندل, تیندال \|
	\| `یناق` \| 2.07x \| 27 contexts \| ایناق, قیناق, سیناق \|
	\| `ئرلش` \| 2.13x \| 24 contexts \| یئرلشن, يئرلشن, یئرلشه \|
	\| `ریخی` \| 2.00x \| 22 contexts \| مریخی, ریخین, مریخین \|
	\| `قاین` \| 2.31x \| 14 contexts \| قاینا, قاینی, قاینز \|
	\| `هرلر` \| 2.14x \| 17 contexts \| شهرلر, شهرلري, شهرلری \|

	### 6.4 Affix Compatibility (Co-occurrence)

	This table shows which prefixes and suffixes most frequently co-occur on the same stems, revealing the 'stacking' rules of the language's morphology.

	No significant affix co-occurrences detected.


	### 6.5 Recursive Morpheme Segmentation

	Using Recursive Hierarchical Substitutability, we decompose complex words into their constituent morphemes. This approach handles nested affixes (e.g., `prefix-prefix-root-suffix`).

	\| Word \| Suggested Split \| Confidence \| Stem \|
	\|------\|-----------------\|------------\|------\|
	\| دۆکانلارینین \| `دۆکانلار-ین-ین` \| 6.0 \| `دۆکانلار` \|
	\| سیلاحینین \| `سیلاح-ین-ین` \| 6.0 \| `سیلاح` \|
	\| باشچی‌لارینین \| `باشچی‌لار-ین-ین` \| 6.0 \| `باشچی‌لار` \|
	\| دوخالارین \| `دوخالار-ین` \| 4.5 \| `دوخالار` \|
	\| آمارلارین \| `آمارلار-ین` \| 4.5 \| `آمارلار` \|
	\| تاریخچیلرین \| `تاریخچیلر-ین` \| 4.5 \| `تاریخچیلر` \|
	\| اؤدوللرین \| `اؤدوللر-ین` \| 4.5 \| `اؤدوللر` \|
	\| شکیلچیلرین \| `شکیلچیلر-ین` \| 4.5 \| `شکیلچیلر` \|
	\| کوْمونیستلرین \| `کوْمونیستلر-ین` \| 4.5 \| `کوْمونیستلر` \|
	\| بیوفیزیکین \| `بیوفیزیک-ین` \| 4.5 \| `بیوفیزیک` \|
	\| تاپینتیلارین \| `تاپینتیلار-ین` \| 4.5 \| `تاپینتیلار` \|
	\| میکروبلارین \| `میکروبلار-ین` \| 4.5 \| `میکروبلار` \|
	\| تیکیلینین \| `تیکیل-ین-ین` \| 3.0 \| `تیکیل` \|
	\| نفتالینین \| `نفتال-ین-ین` \| 3.0 \| `نفتال` \|
	\| والنتاینین \| `والنتا-ین-ین` \| 3.0 \| `والنتا` \|

	### 6.6 Linguistic Interpretation

	> Automated Insight:
	The language South Azerbaijani shows high morphological productivity. The subword models are significantly more efficient than word models, suggesting a rich system of affixation or compounding.

	---
	## 7. Summary & Recommendations

	![Performance Dashboard](visualizations/performance_dashboard.png)

	### Production Recommendations

	\| Component \| Recommended \| Rationale \|
	\|-----------\|-------------\|-----------\|
	\| Tokenizer \| 64k BPE \| Best compression (4.15x) \|
	\| N-gram \| 2-gram \| Lowest perplexity (528) \|
	\| Markov \| Context-4 \| Highest predictability (96.6%) \|
	\| Embeddings \| 100d \| Balanced semantic capture and isotropy \|


	---
	## Appendix: Metrics Glossary & Interpretation Guide

	This section provides definitions, intuitions, and guidance for interpreting the metrics used throughout this report.

	### Tokenizer Metrics

	Compression Ratio
	> Definition: The ratio of characters to tokens (chars/token). Measures how efficiently the tokenizer represents text.
	>
	> Intuition: Higher compression means fewer tokens needed to represent the same text, reducing sequence lengths for downstream models. A 3x compression means ~3 characters per token on average.
	>
	> What to seek: Higher is generally better for efficiency, but extremely high compression may indicate overly aggressive merging that loses morphological information.

	Average Token Length (Fertility)
	> Definition: Mean number of characters per token produced by the tokenizer.
	>
	> Intuition: Reflects the granularity of tokenization. Longer tokens capture more context but may struggle with rare words; shorter tokens are more flexible but increase sequence length.
	>
	> What to seek: Balance between 2-5 characters for most languages. Arabic/morphologically-rich languages may benefit from slightly longer tokens.

	Unknown Token Rate (OOV Rate)
	> Definition: Percentage of tokens that map to the unknown/UNK token, indicating words the tokenizer cannot represent.
	>
	> Intuition: Lower OOV means better vocabulary coverage. High OOV indicates the tokenizer encounters many unseen character sequences.
	>
	> What to seek: Below 1% is excellent; below 5% is acceptable. BPE tokenizers typically achieve very low OOV due to subword fallback.

	### N-gram Model Metrics

	Perplexity
	> Definition: Measures how "surprised" the model is by test data. Mathematically: 2^(cross-entropy). Lower values indicate better prediction.
	>
	> Intuition: If perplexity is 100, the model is as uncertain as if choosing uniformly among 100 options at each step. A perplexity of 10 means effectively choosing among 10 equally likely options.
	>
	> What to seek: Lower is better. Perplexity decreases with larger n-grams (more context). Values vary widely by language and corpus size.

	Entropy
	> Definition: Average information content (in bits) needed to encode the next token given the context. Related to perplexity: perplexity = 2^entropy.
	>
	> Intuition: High entropy means high uncertainty/randomness; low entropy means predictable patterns. Natural language typically has entropy between 1-4 bits per character.
	>
	> What to seek: Lower entropy indicates more predictable text patterns. Entropy should decrease as n-gram size increases.

	Coverage (Top-K)
	> Definition: Percentage of corpus occurrences explained by the top K most frequent n-grams.
	>
	> Intuition: High coverage with few patterns indicates repetitive/formulaic text; low coverage suggests diverse vocabulary usage.
	>
	> What to seek: Depends on use case. For language modeling, moderate coverage (40-60% with top-1000) is typical for natural text.

	### Markov Chain Metrics

	Average Entropy
	> Definition: Mean entropy across all contexts, measuring average uncertainty in next-word prediction.
	>
	> Intuition: Lower entropy means the model is more confident about what comes next. Context-1 has high entropy (many possible next words); Context-4 has low entropy (few likely continuations).
	>
	> What to seek: Decreasing entropy with larger context sizes. Very low entropy (<0.1) indicates highly deterministic transitions.

	Branching Factor
	> Definition: Average number of unique next tokens observed for each context.
	>
	> Intuition: High branching = many possible continuations (flexible but uncertain); low branching = few options (predictable but potentially repetitive).
	>
	> What to seek: Branching factor should decrease with context size. Values near 1.0 indicate nearly deterministic chains.

	Predictability
	> Definition: Derived metric: (1 - normalized_entropy) × 100%. Indicates how deterministic the model's predictions are.
	>
	> Intuition: 100% predictability means the next word is always certain; 0% means completely random. Real text falls between these extremes.
	>
	> What to seek: Higher predictability for text generation quality, but too high (>98%) may produce repetitive output.

	### Vocabulary & Zipf's Law Metrics

	Zipf's Coefficient
	> Definition: The slope of the log-log plot of word frequency vs. rank. Zipf's law predicts this should be approximately -1.
	>
	> Intuition: A coefficient near -1 indicates the corpus follows natural language patterns where a few words are very common and most words are rare.
	>
	> What to seek: Values between -0.8 and -1.2 indicate healthy natural language distribution. Deviations may suggest domain-specific or artificial text.

	R² (Coefficient of Determination)
	> Definition: Measures how well the linear fit explains the frequency-rank relationship. Ranges from 0 to 1.
	>
	> Intuition: R² near 1.0 means the data closely follows Zipf's law; lower values indicate deviation from expected word frequency patterns.
	>
	> What to seek: R² > 0.95 is excellent; > 0.99 indicates near-perfect Zipf adherence typical of large natural corpora.

	Vocabulary Coverage
	> Definition: Cumulative percentage of corpus tokens accounted for by the top N words.
	>
	> Intuition: Shows how concentrated word usage is. If top-100 words cover 50% of text, the corpus relies heavily on common words.
	>
	> What to seek: Top-100 covering 30-50% is typical. Higher coverage indicates more repetitive text; lower suggests richer vocabulary.

	### Word Embedding Metrics

	Isotropy
	> Definition: Measures how uniformly distributed vectors are in the embedding space. Computed as the ratio of minimum to maximum singular values.
	>
	> Intuition: High isotropy (near 1.0) means vectors spread evenly in all directions; low isotropy means vectors cluster in certain directions, reducing expressiveness.
	>
	> What to seek: Higher isotropy generally indicates better-quality embeddings. Values > 0.1 are reasonable; > 0.3 is good. Lower-dimensional embeddings tend to have higher isotropy.

	Average Norm
	> Definition: Mean magnitude (L2 norm) of word vectors in the embedding space.
	>
	> Intuition: Indicates the typical "length" of vectors. Consistent norms suggest stable training; high variance may indicate some words are undertrained.
	>
	> What to seek: Relatively consistent norms across models. The absolute value matters less than consistency (low std deviation).

	Cosine Similarity
	> Definition: Measures angular similarity between vectors, ranging from -1 (opposite) to 1 (identical direction).
	>
	> Intuition: Words with similar meanings should have high cosine similarity. This is the standard metric for semantic relatedness in embeddings.
	>
	> What to seek: Semantically related words should score > 0.5; unrelated words should be near 0. Synonyms often score > 0.7.

	t-SNE Visualization
	> Definition: t-Distributed Stochastic Neighbor Embedding - a dimensionality reduction technique that preserves local structure for visualization.
	>
	> Intuition: Clusters in t-SNE plots indicate groups of semantically related words. Spread indicates vocabulary diversity; tight clusters suggest semantic coherence.
	>
	> What to seek: Meaningful clusters (e.g., numbers together, verbs together). Avoid over-interpreting distances - t-SNE preserves local, not global, structure.

	### General Interpretation Guidelines

	1. Compare within model families: Metrics are most meaningful when comparing models of the same type (e.g., 8k vs 64k tokenizer).
	2. Consider trade-offs: Better performance on one metric often comes at the cost of another (e.g., compression vs. OOV rate).
	3. Context matters: Optimal values depend on downstream tasks. Text generation may prioritize different metrics than classification.
	4. Corpus influence: All metrics are influenced by corpus characteristics. Wikipedia text differs from social media or literature.
	5. Language-specific patterns: Morphologically rich languages (like Arabic) may show different optimal ranges than analytic languages.


	### Visualizations Index

	\| Visualization \| Description \|
	\|---------------\|-------------\|
	\| Tokenizer Compression \| Compression ratios by vocabulary size \|
	\| Tokenizer Fertility \| Average token length by vocabulary \|
	\| Tokenizer OOV \| Unknown token rates \|
	\| Tokenizer Total Tokens \| Total tokens by vocabulary \|
	\| N-gram Perplexity \| Perplexity by n-gram size \|
	\| N-gram Entropy \| Entropy by n-gram size \|
	\| N-gram Coverage \| Top pattern coverage \|
	\| N-gram Unique \| Unique n-gram counts \|
	\| Markov Entropy \| Entropy by context size \|
	\| Markov Branching \| Branching factor by context \|
	\| Markov Contexts \| Unique context counts \|
	\| Zipf's Law \| Frequency-rank distribution with fit \|
	\| Vocab Frequency \| Word frequency distribution \|
	\| Top 20 Words \| Most frequent words \|
	\| Vocab Coverage \| Cumulative coverage curve \|
	\| Embedding Isotropy \| Vector space uniformity \|
	\| Embedding Norms \| Vector magnitude distribution \|
	\| Embedding Similarity \| Word similarity heatmap \|
	\| Nearest Neighbors \| Similar words for key terms \|
	\| t-SNE Words \| 2D word embedding visualization \|
	\| t-SNE Sentences \| 2D sentence embedding visualization \|
	\| Position Encoding \| Encoding method comparison \|
	\| Model Sizes \| Storage requirements \|
	\| Performance Dashboard \| Comprehensive performance overview \|

	---
	## About This Project

	### Data Source

	Models trained on [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly) - a monthly snapshot of Wikipedia articles across 300+ languages.

	### Project

	A project by [Wikilangs](https://wikilangs.org) - Open-source NLP models for every Wikipedia language.

	### Maintainer

	[Omar Kamali](https://omarkamali.com) - [Omneity Labs](https://omneitylabs.com)

	### Citation

	If you use these models in your research, please cite:

	```bibtex
	@misc{wikilangs2025,
	author = {Kamali, Omar},
	title = {Wikilangs: Open NLP Models for Wikipedia Languages},
	year = {2025},
	doi = {10.5281/zenodo.18073153},
	publisher = {Zenodo},
	url = {https://huggingface.co/wikilangs}
	institution = {Omneity Labs}
	}
	```

	### License

	MIT License - Free for academic and commercial use.

	### Links

	- 🌐 Website: [wikilangs.org](https://wikilangs.org)
	- 🤗 Models: [huggingface.co/wikilangs](https://huggingface.co/wikilangs)
	- 📊 Data: [wikipedia-monthly](https://huggingface.co/datasets/omarkamali/wikipedia-monthly)
	- 👤 Author: [Omar Kamali](https://huggingface.co/omarkamali)
	- 🤝 Sponsor: [Featherless AI](https://featherless.ai)
	---
	Generated by Wikilangs Models Pipeline

	Report Date: 2026-01-03 19:16:27