readme: ---
  language: 
  - bn
  - en
  licenses:
  - cc-by-nc-sa-4.0
  ---
  
  # BanglishBERT
  
  This repository contains the pretrained discriminator checkpoint of the model **BanglishBERT**. This is an [ELECTRA](https://openreview.net/pdf?id=r1xMH1BtvB) discriminator model pretrained with the Replaced Token Detection (RTD) objective on large amounts of Bengali and English corpora. BanglishBERT achieves state-of-the-art **zero-shot cross-lingual transfer** results in many of the NLP tasks in Bengali. 
  
  For finetuning on different downstream tasks such as `Sentiment classification`, `Named Entity Recognition`, `Natural Language Inference` etc., refer to the scripts in the official GitHub [repository](https://github.com/csebuetnlp/banglabert).
  
  **Note**: This model was pretrained using a specific normalization pipeline available [here](https://github.com/csebuetnlp/normalizer). All finetuning scripts in the official GitHub repository uses this normalization by default. If you need to adapt the pretrained model for a different task make sure the text units are normalized using this pipeline before tokenizing to get best results. A basic example is given below:
  
  ## Using this model as a discriminator in `transformers` (tested on 4.11.0.dev0)
  
  ```python
  from transformers import AutoModelForPreTraining, AutoTokenizer
  from normalizer import normalize # pip install git+https://github.com/csebuetnlp/normalizer
  import torch
  
  model = AutoModelForPreTraining.from_pretrained("csebuetnlp/banglishbert")
  tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglishbert")
  
  original_sentence = "আমি কৃতজ্ঞ কারণ আপনি আমার জন্য অনেক কিছু করেছেন।"
  fake_sentence = "আমি হতাশ কারণ আপনি আমার জন্য অনেক কিছু করেছেন।"
  fake_sentence = normalize(fake_sentence) # this normalization step is required before tokenizing the text
  
  fake_tokens = tokenizer.tokenize(fake_sentence)
  fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
  discriminator_outputs = model(fake_inputs).logits
  predictions = torch.round((torch.sign(discriminator_outputs) + 1) / 2)
  
  [print("%7s" % token, end="") for token in fake_tokens]
  print("\n" + "-" * 50)
  [print("%7s" % int(prediction), end="") for prediction in predictions.squeeze().tolist()[1:-1]]
  print("\n" + "-" * 50)
  ```
  
  ## Benchmarks
   
  * Zero-shot cross-lingual transfer-learning
  
  |     Model          |   Params   |     SC (macro-F1)     |      NLI (accuracy)     |    NER  (micro-F1)   |   QA (EM/F1)   |   BangLUE score |
  |----------------|-----------|-----------|-----------|-----------|-----------|-----------|
  |[mBERT](https://huggingface.co/bert-base-multilingual-cased) | 180M  | 27.05 | 62.22 | 39.27 | 59.01/64.18 |  50.35 |
  |[XLM-R (base)](https://huggingface.co/xlm-roberta-base) |  270M   | 42.03 | 72.18 | 45.37 | 55.03/61.83 |  55.29 |
  |[XLM-R (large)](https://huggingface.co/xlm-roberta-large) | 550M  | 49.49 | 78.13 | 56.48 | 71.13/77.70 |  66.59 |
  |[BanglishBERT](https://huggingface.co/csebuetnlp/banglishbert) | 110M | 48.39 | 75.26 | 55.56 | 72.87/78.63 | 66.14 |
  
  * Supervised fine-tuning
  
  |     Model          |   Params   |     SC (macro-F1)     |      NLI (accuracy)     |    NER  (micro-F1)   |   QA (EM/F1)   |   BangLUE score |
  |----------------|-----------|-----------|-----------|-----------|-----------|-----------|
  |[mBERT](https://huggingface.co/bert-base-multilingual-cased) | 180M  | 67.59 | 75.13 | 68.97 | 67.12/72.64 | 70.29 |
  |[XLM-R (base)](https://huggingface.co/xlm-roberta-base) |  270M   | 69.54 | 78.46 | 73.32 | 68.09/74.27  | 72.82 |        
  |[XLM-R (large)](https://huggingface.co/xlm-roberta-large) | 550M  | 70.97 | 82.40 | 78.39 | 73.15/79.06 | 76.79 |
  |[sahajBERT](https://huggingface.co/neuropark/sahajBERT) | 18M | 71.12 | 76.92 | 70.94 | 65.48/70.69 | 71.03 |
  |[BanglishBERT](https://huggingface.co/csebuetnlp/banglishbert) | 110M | 70.61 | 80.95 | 76.28 | 72.43/78.40 | 75.73 |
  |[BanglaBERT](https://huggingface.co/csebuetnlp/banglabert) | 110M | 72.89 | 82.80 | 77.78 | 72.63/79.34 | **77.09** |
  
  
  
  
  The benchmarking datasets are as follows:
  * **SC:** **[Sentiment Classification](https://aclanthology.org/2021.findings-emnlp.278)**
  * **NER:** **[Named Entity Recognition](https://multiconer.github.io/competition)**
  * **NLI:** **[Natural Language Inference](https://github.com/csebuetnlp/banglabert/#datasets)**
  * **QA:** **[Question Answering](https://github.com/csebuetnlp/banglabert/#datasets)**
  
  ## Citation
  
  If you use this model, please cite the following paper:
  ```
  @inproceedings{bhattacharjee-etal-2022-banglabert,
      title     = {BanglaBERT: Lagnuage Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla},
      author = "Bhattacharjee, Abhik  and
        Hasan, Tahmid  and
        Mubasshir, Kazi  and
        Islam, Md. Saiful  and
        Uddin, Wasi Ahmad  and
        Iqbal, Anindya  and
        Rahman, M. Sohel  and
        Shahriyar, Rifat",
        booktitle = "Findings of the North American Chapter of the Association for Computational Linguistics: NAACL 2022",
        mo