readme: # <a name="introduction"></a> PhoBERT: Pre-trained language models for Vietnamese 
    
  Pre-trained PhoBERT models are the state-of-the-art language models for Vietnamese ([Pho](https://en.wikipedia.org/wiki/Pho), i.e. "Phở", is a popular food in Vietnam):
  
   - Two PhoBERT versions of "base" and "large" are the first public large-scale monolingual language models pre-trained for Vietnamese. PhoBERT pre-training approach is based on [RoBERTa](https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.md)  which optimizes the [BERT](https://github.com/google-research/bert) pre-training procedure for more robust performance.
   - PhoBERT outperforms previous monolingual and multilingual approaches, obtaining new state-of-the-art performances on four downstream Vietnamese NLP tasks of Part-of-speech tagging, Dependency parsing, Named-entity recognition and Natural language inference.
  
  The general architecture and experimental results of PhoBERT can be found in our EMNLP-2020 Findings [paper](https://arxiv.org/abs/2003.00744):
  
      @article{phobert,
      title     = {{PhoBERT: Pre-trained language models for Vietnamese}},
      author    = {Dat Quoc Nguyen and Anh Tuan Nguyen},
      journal   = {Findings of EMNLP},
      year      = {2020}
      }
  
  **Please CITE** our paper when PhoBERT is used to help produce published results or is incorporated into other software.
  
  For further information or requests, please go to [PhoBERT's homepage](https://github.com/VinAIResearch/PhoBERT)!
  
  