readme: ---
  language:
  - as
  - bn
  - gu
  - hi
  - mr
  - ne
  - or
  - pa
  - si
  - sa
  - bpy
  - mai
  - bh
  - gom
  license: apache-2.0
  datasets:
  - oscar
  tags:
  - multilingual
  - albert
  - masked-language-modeling
  - sentence-order-prediction
  - fill-mask
  - xlmindic
  - nlp
  - indoaryan
  - indicnlp
  - iso15919
  - transliteration
  widget:
  - text : 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli [MASK], aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika. 1913 sālē gītāñjali kābyagranthēra iṁrēji anubādēra janya tini ēśīẏadēra madhyē sāhityē prathama nōbēla puraskāra lābha karēna.'
  
  co2_eq_emissions:
        emissions: 28.53
        source: "calculated using this webstie https://mlco2.github.io/impact/#compute"
        training_type: "pretraining"
        geographical_location: "NA"
        hardware_used: "TPUv3-8 for about 180 hours or 7.5 days"
  ---
  
  # XLMIndic Base Uniscript
  
  This model is pretrained on a subset of the [OSCAR](https://huggingface.co/datasets/oscar) corpus spanning 14 Indo-Aryan languages. **Before pretraining this model we transliterate the text to [ISO-15919](https://en.wikipedia.org/wiki/ISO_15919) format using the [Aksharamukha](https://pypi.org/project/aksharamukha/)
  library.** A demo of Aksharamukha library is hosted [here](https://aksharamukha.appspot.com/converter)
  where you can transliterate your text and use it on our model on the inference widget.
  
  ## Model description
  
  This model has the same configuration as the [ALBERT Base v2 model](https://huggingface.co/albert-base-v2/). Specifically, this model has the following configuration:
  
  - 12 repeating layers
  - 128 embedding dimension
  - 768 hidden dimension
  - 12 attention heads
  - 11M parameters
  - 512 sequence length
  
  ## Training data
  
  This model was pretrained on the [OSCAR](https://huggingface.co/datasets/oscar) dataset which is a medium sized multilingual corpus containing text from 163 languages. We select a subset of 14 languages based on the following criteria:
   - Belongs to the [Indo-Aryan language family](https://en.wikipedia.org/wiki/Indo-Aryan_languages).
   - Uses a [Brahmic script](https://en.wikipedia.org/wiki/Brahmic_scripts).
   
  These are the 14 languages we pretrain this model on:
  - Assamese
  - Bangla
  - Bihari
  - Bishnupriya Manipuri
  - Goan Konkani
  - Gujarati
  - Hindi
  - Maithili
  - Marathi
  - Nepali
  - Oriya
  - Panjabi
  - Sanskrit
  - Sinhala
  
  ## Transliteration
  
  *The unique component of this model is that it takes in ISO-15919 transliterated text.*
  
  The motivation behind this is this. When two languages share vocabularies, a machine learning model can exploit that to learn good cross-lingual representations. However if these two languages use different writing scripts it is difficult for a model to make the connection. Thus if if we can write the two languages in a single script then it is easier for the model to learn good cross-lingual representation.
  
  For many of the scripts currently in use, there are standard transliteration schemes to convert to the Latin script. In particular, for the Indic scripts the ISO-15919 transliteration scheme is designed to consistently transliterate texts written in different Indic scripts to the Latin script.
  
  An example of ISO-15919 transliteration for a piece of **Bangla** text is the following:
  
  **Original:** "রবীন্দ্রনাথ ঠাকুর এফআরএএস (৭ মে ১৮৬১ - ৭ আগস্ট ১৯৪১; ২৫ বৈশাখ ১২৬৮ - ২২ শ্রাবণ ১৩৪৮ বঙ্গাব্দ) ছিলেন অগ্রণী বাঙালি কবি, ঔপন্যাসিক, সংগীতস্রষ্টা, নাট্যকার, চিত্রকর, ছোটগল্পকার, প্রাবন্ধিক, অভিনেতা, কণ্ঠশিল্পী ও দার্শনিক।"
  
  **Transliterated:** 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli kabi, aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika.'
  
  Another example for a piece of **Hindi** text is the following:
  
  **Original:** "चूंकि मानव परिवार के सभी सदस्यों के जन्मजात गौरव और समान तथा अविच्छिन्न अधिकार की स्वीकृति ही विश्व-शान्ति, न्याय और स्वतन्त्रता की बुनियाद है"
  
  **Transliterated:** "cūṁki mānava parivāra kē sabhī sadasyōṁ kē janmajāta gaurava aura samāna tathā avicchinna adhikāra kī svīkr̥ti hī viśva-śānti, nyāya aura svatantratā kī buniyāda hai"
   
   
  ## Training procedure
  
  ### Preprocessing
  
  The texts are transliterated to ISO-15919 format using the Aksharamukha library. Then these are tokenized using SentencePiece and a vocabulary size of 50,000. The inputs of the model are
  then of the form:
  ```
  [CLS] Sentence A [SEP] Sentence B [SEP]
  ```
  
  ### Training
  
  Training objective is the same as the original ALBERT. 
  .
  The details of the masking procedure for each sentence are the following:
  - 15% of the tokens are masked.
  - In 80% of the cases, the masked tokens are replaced by `[MASK]`.
  - In 10% of the cases, the masked tokens are replaced by a random token (different) from the one they replace.
  - In the 10% remaining cases, the masked tokens are left as is