readme: ---
  language: pl
  ---
  
  # HerBERT 
  **[HerBERT](https://en.wikipedia.org/wiki/Zbigniew_Herbert)** is a BERT-based Language Model trained on Polish Corpora
  using only MLM objective with dynamic masking of whole words. For more details, please refer to: 
  [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://arxiv.org/abs/2005.00630).
  
  ## Dataset
  **HerBERT** training dataset is a combination of several publicly available corpora for Polish language:
  
  | Corpus | Tokens | Texts |
  | :------ | ------: | ------: |
  | [OSCAR](https://traces1.inria.fr/oscar/)| 6710M  | 145M |
  | [Open Subtitles](http://opus.nlpl.eu/OpenSubtitles-v2018.php) | 1084M  | 1.1M |
  | [Wikipedia](https://dumps.wikimedia.org/) | 260M  | 1.5M |
  | [Wolne Lektury](https://wolnelektury.pl/) | 41M  | 5.5k |
  | [Allegro Articles](https://allegro.pl/artykuly) | 18M  | 33k |
  
  ## Tokenizer
  The training dataset was tokenized into subwords using [HerBERT Tokenizer](https://huggingface.co/allegro/herbert-klej-cased-tokenizer-v1); a character level byte-pair encoding with
  a vocabulary size of 50k tokens. The tokenizer itself was trained on [Wolne Lektury](https://wolnelektury.pl/) and a publicly available subset of 
  [National Corpus of Polish](http://nkjp.pl/index.php?page=14&lang=0) with a [fastBPE](https://github.com/glample/fastBPE) library.
  
  Tokenizer utilizes `XLMTokenizer` implementation for that reason, one should load it as `allegro/herbert-klej-cased-tokenizer-v1`.
  
  ## HerBERT models summary
  | Model | WWM | Cased | Tokenizer | Vocab Size  | Batch Size | Train Steps |
  | :------ | ------: | ------: | ------: | ------: | ------: | ------: |
  | herbert-klej-cased-v1 | YES | YES | BPE | 50K | 570 | 180k | 
  
  ## Model evaluation
  HerBERT was evaluated on the [KLEJ](https://klejbenchmark.com/) benchmark, publicly available set of nine evaluation tasks for the Polish language understanding.
  It had the best average performance and obtained the best results for three of them.
  
  | Model | Average | NKJP-NER | CDSC-E | CDSC-R | CBD | PolEmo2.0-IN\t|PolEmo2.0-OUT | DYK | PSC | AR\t|
  | :------ | ------: | ------: | ------: | ------: | ------: | ------: | ------: |  ------: | ------: | ------: |
  | herbert-klej-cased-v1 | **80.5** | 92.7 | 92.5 | 91.9 | **50.3** | **89.2** |**76.3** |52.1 |95.3 | 84.5 |
  
  Full leaderboard is available [online](https://klejbenchmark.com/leaderboard). 
  
  
  ## HerBERT usage
  Model training and experiments were conducted with [transformers](https://github.com/huggingface/transformers) in version 2.0.
  
  Example code:
  ```python
  from transformers import XLMTokenizer, RobertaModel
  
  tokenizer = XLMTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
  model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")
  
  encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors='pt')
  outputs = model(encoded_input)
  ```
  
  HerBERT can also be loaded using `AutoTokenizer` and `AutoModel`:
  
  ```python
  tokenizer = AutoTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
  model = AutoModel.from_pretrained("allegro/herbert-klej-cased-v1")
  ```
  
  ## License
  CC BY-SA 4.0
  
  ## Citation
  If you use this model, please cite the following paper:
  ```
  @inproceedings{rybak-etal-2020-klej,
      title = "{KLEJ}: Comprehensive Benchmark for {P}olish Language Understanding",
      author = "Rybak, Piotr  and
        Mroczkowski, Robert  and
        Tracz, Janusz  and
        Gawlik, Ireneusz",
      booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
      month = jul,
      year = "2020",
      address = "Online",
      publisher = "Association for Computational Linguistics",
      url = "https://www.aclweb.org/anthology/2020.acl-main.111",
      doi = "10.18653/v1/2020.acl-main.111",
      pages = "1191--1201",
  }
  ```
  
  ## Authors
  The model was trained by **Allegro Machine Learning Research** team.
  
  You can contact us at: <a href="mailto:klejbenchmark@allegro.pl">klejbenchmark@allegro.pl</a>
  