readme: ---
  language: 
  - bn
  licenses:
  - cc-by-nc-sa-4.0
  ---
  
  # BanglaT5
  
  This repository contains the pretrained checkpoint of the model **BanglaT5**. This is a sequence to sequence transformer model pretrained with the ["Span Corruption"]() objective. Finetuned models using this checkpoint achieve state-of-the-art results on many of the NLG tasks in bengali. 
  
  For finetuning on different downstream tasks such as `Machine Translation`, `Abstractive Text Summarization`, `Question Answering` etc., refer to the scripts in the official GitHub [repository](https://github.com/csebuetnlp/BanglaNLG).
  
  **Note**: This model was pretrained using a specific normalization pipeline available [here](https://github.com/csebuetnlp/normalizer). All finetuning scripts in the official GitHub repository use this normalization by default. If you need to adapt the pretrained model for a different task make sure the text units are normalized using this pipeline before tokenizing to get best results. A basic example is given below:
  
  ## Using this model in `transformers` (tested on 4.11.0.dev0)
  
  ```python
  from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
  from normalizer import normalize # pip install git+https://github.com/csebuetnlp/normalizer
  
  model = AutoModelForSeq2SeqLM.from_pretrained("csebuetnlp/banglat5")
  tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglat5", use_fast=False)
  
  input_sentence = ""
  input_ids = tokenizer(normalize(input_sentence), return_tensors="pt").input_ids
  generated_tokens = model.generate(input_ids)
  decoded_tokens = tokenizer.batch_decode(generated_tokens)[0]
  
  print(decoded_tokens)
  ```
  
  ## Benchmarks
   
  * Supervised fine-tuning
  
  |     Model          |   Params   |     MT (SacreBLEU)    |      TS (ROUGE-2)     |      QA (EM/F1)   |   MD (SacreBLEU-1)  |  NHG (ROUGE-2) |  XLS (ROUGE-2) |   BNLG score |
  |--------------------|------------|-----------------------|------------------------|-------------------|--------------------|----------------|----------------|---------------|
  |[mT5 (base)](https://huggingface.co/google/mt5-base) | 582M  | 36.6/22.5 | 10.3 | 59.0/65.3 | 17.5 |  9.6 | 2.7/0.7 | 24.9 |
  |[XLM-ProphetNet](https://huggingface.co/microsoft/xprophetnet-large-wiki100-cased) | 616M | 23.3/16.4 | 7.8 | 53.0/57.3 | 20.0 | 9.5 | 6.2/2.7 | 21.8 |
  |[mBART-50](https://huggingface.co/facebook/mbart-large-50) | 611M | 23.6/16.7 | 10.4 | 53.4/58.9 | 18.5 | 11.2 | 5.4/3.7 | 22.4 |
  |[IndicBART](https://huggingface.co/ai4bharat/IndicBART) | 244M | 22.7/13.1 | 8.1 | 53.3/58.8 | 14.8 | 7.9 | 6.3/2.5 | 20.8 |
  |[BanglaT5](https://huggingface.co/csebuetnlp/banglat5) | 247M | 38.8/25.2 | 13.7 | 68.5/74.8 | 19.0 | 13.8 | 6.4/4.0 | 29.4 |
  
  
  The benchmarking datasets are as follows:
  * **MT:** **[Machine Translation](https://github.com/csebuetnlp/banglanmt#datasets)**
  * **TS:** **[Abstractive Text Summarization](https://huggingface.co/datasets/csebuetnlp/xlsum)**
  * **QA:** **[Question Answering](https://huggingface.co/datasets/csebuetnlp/squad_bn)**
  * **MD:** **[Multi Turn Dialogue Generation](https://drive.google.com/file/d/1qPmNN6qA4evbh4cD_BDDTCFOwMu4H2JS/view?usp=sharing)**
  * **NHG:** **[News Headline Generation](https://huggingface.co/datasets/csebuetnlp/xlsum)**
  * **XLS:** **[Cross-lingual Summarization](https://huggingface.co/datasets/csebuetnlp/CrossSum)**
  
  ## Citation
  
  If you use this model, please cite the following paper:
  ```
  @article{bhattacharjee2022banglanlg,
    author    = {Abhik Bhattacharjee and Tahmid Hasan and Wasi Uddin Ahmad and Rifat Shahriyar},
    title     = {BanglaNLG: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla},
    journal   = {CoRR},
    volume    = {abs/2205.11081},
    year      = {2022},
    url       = {https://arxiv.org/abs/2205.11081},
    eprinttype = {arXiv},
    eprint    = {2205.11081}
  }
  ```
  
  
  If you use the normalization module, please cite the following paper:
  ```
  @inproceedings{hasan-etal-2020-low,
      title = "Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for {B}engali-{E}nglish Machine Translation",
      author = "Hasan, Tahmid  and
        Bhattacharjee, Abhik  and
        Samin, Kazi  and
        Hasan, Masum  and
        Basak, Madhusudan  and
        Rahman, M. Sohel  and
        Shahriyar, Rifat",
      booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
      month = nov,
      year = "2020",
      address = "Online",
      publisher = "Association for Computational Linguistics",
      url = "https://www.aclweb.org/anthology/2020.emnlp-main.207",
      doi = "10.18653/v1/2020.emnlp-main.207",
      pages = "2612--2623",
      abstract = "Despite being the seventh most widely spoken language in the world, Bengali has received much less attention in machine translation literature due to being low in resources. Most publicly available parallel corpora for Bengali are not large enough; and have rather poor quality, mostly because of incorrect sentence alignments resulting f