readme: ---
  language:
      - bn
  thumbnail:
  tags:
      -
  license: apache-2.0
  datasets:
      - oscar
      - wikipedia
  metrics:
      -
  ---
  
  # [WIP] Albert Bengali - dev version
  
  ## Model description
  
  For the moment, only the tokenizer is available. The tokenizer is based on [SentencePiece](https://github.com/google/sentencepiece) with Unigram language model segmentation algorithm.
  
  Taking into account certain characteristics of the language, we chose that:
  
  -   the tokenizer passes in lower case all the texts because the Bengali language is a monocameral scrip (no difference between capital and lower case);
  -   the sentence pieces can't go beyond the boundary of a word because the words are spaced by white spaces in the Bengali language.
  
  ## Intended uses & limitations
  
  This tokenizer is adapted to the Bengali language. You can use it to pre-train an Albert model on the Bengali language.
  
  #### How to use
  
  To tokenize:
  
  ```python
  from transformers import AlbertTokenizer
  tokenizer = AlbertTokenizer.from_pretrained('SaulLu/albert-bn-dev')
  text = "পোকেমন  জাপানী ভিডিও গেম কোম্পানি নিনটেন্ডো কর্তৃক প্রকাশিত একটি মিডিয়া ফ্র‍্যাঞ্চাইজি।"
  encoded_input = tokenizer(text, return_tensors='pt')
  ```
  
  #### Limitations and bias
  
  Provide examples of latent issues and potential remediations.
  
  ## Training data
  
  The tokenizer was trained on a random subset of 4M sentences of Bengali Oscar and Bengali Wikipedia.
  
  ## Training procedure
  
  ### Tokenizer
  
  The tokenizer was trained with the [SentencePiece](https://github.com/google/sentencepiece) on 8 x Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz with 16GB RAM and 36GB SWAP.
  
  ```
  import sentencepiece as spm
  config = {
      "input": "./dataset/oscar_bn.txt,./dataset/wikipedia_bn.txt",
      "input_format": "text",
      "model_type": "unigram",
      "vocab_size": 32000,
      "self_test_sample_size": 0,
      "character_coverage": 0.9995,
      "shuffle_input_sentence": true,
      "seed_sentencepiece_size": 1000000,
      "shrinking_factor": 0.75,
      "num_threads": 8,
      "num_sub_iterations": 2,
      "max_sentencepiece_length": 16,
      "max_sentence_length": 4192,
      "split_by_unicode_script": true,
      "split_by_number": true,
      "split_digits": true,
      "control_symbols": "[MASK]",
      "byte_fallback": false,
      "vocabulary_output_piece_score": true,
      "normalization_rule_name": "nmt_nfkc_cf",
      "add_dummy_prefix": true,
      "remove_extra_whitespaces": true,
      "hard_vocab_limit": true,
      "unk_id": 1,
      "bos_id": 2,
      "eos_id": 3,
      "pad_id": 0,
      "bos_piece": "[CLS]",
      "eos_piece": "[SEP]",
      "train_extremely_large_corpus": true,
      "split_by_whitespace": true,
      "model_prefix": "./spiece",
      "input_sentence_size": 4000000,
      "user_defined_symbols": "(,),-,.,–,£,।",
  }
  spm.SentencePieceTrainer.train(**config)
  ```
  
  <!-- ## Eval results
  
  ### BibTeX entry and citation info
  
  ```bibtex
  @inproceedings{...,
    year={2020}
  }
  ``` -->
  