readme: ---
  language: pl
  tags:
  - herbert
  license: cc-by-4.0
  ---
  
  # HerBERT 
  **[HerBERT](https://en.wikipedia.org/wiki/Zbigniew_Herbert)** is a BERT-based Language Model trained on Polish corpora
  using Masked Language Modelling (MLM) and Sentence Structural Objective (SSO) with dynamic masking of whole words. For more details, please refer to: [HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish](https://www.aclweb.org/anthology/2021.bsnlp-1.1/).
  
  Model training and experiments were conducted with [transformers](https://github.com/huggingface/transformers) in version 2.9.
  
  ## Corpus
  HerBERT was trained on six different corpora available for Polish language:
  
  | Corpus | Tokens | Documents |
  | :------ | ------: | ------: |
  | [CCNet Middle](https://github.com/facebookresearch/cc_net) | 3243M  | 7.9M |
  | [CCNet Head](https://github.com/facebookresearch/cc_net) | 2641M  | 7.0M |
  | [National Corpus of Polish](http://nkjp.pl/index.php?page=14&lang=1)| 1357M  | 3.9M |
  | [Open Subtitles](http://opus.nlpl.eu/OpenSubtitles-v2018.php) | 1056M  | 1.1M 
  | [Wikipedia](https://dumps.wikimedia.org/) | 260M  | 1.4M |
  | [Wolne Lektury](https://wolnelektury.pl/) | 41M  | 5.5k |
  
  ## Tokenizer
  The training dataset was tokenized into subwords using a character level byte-pair encoding (``CharBPETokenizer``) with
  a vocabulary size of 50k tokens. The tokenizer itself was trained with a [tokenizers](https://github.com/huggingface/tokenizers) library. 
  
  We kindly encourage you to use the ``Fast`` version of the tokenizer, namely ``HerbertTokenizerFast``.
  
  ## Usage
  Example code:
  ```python
  from transformers import AutoTokenizer, AutoModel
  
  tokenizer = AutoTokenizer.from_pretrained("allegro/herbert-large-cased")
  model = AutoModel.from_pretrained("allegro/herbert-large-cased")
  
  output = model(
      **tokenizer.batch_encode_plus(
          [
              (
                  "A potem szedł środkiem drogi w kurzawie, bo zamiatał nogami, ślepy dziad prowadzony przez tłustego kundla na sznurku.",
                  "A potem leciał od lasu chłopak z butelką, ale ten ujrzawszy księdza przy drodze okrążył go z dala i biegł na przełaj pól do karczmy."
              )
          ],
      padding='longest',
      add_special_tokens=True,
      return_tensors='pt'
      )
  )
  ```
  
  ## License
  CC BY 4.0
  
  ## Citation
  If you use this model, please cite the following paper:
  ```
  @inproceedings{mroczkowski-etal-2021-herbert,
      title = "{H}er{BERT}: Efficiently Pretrained Transformer-based Language Model for {P}olish",
      author = "Mroczkowski, Robert  and
        Rybak, Piotr  and
        Wr{\'o}blewska, Alina  and
        Gawlik, Ireneusz",
      booktitle = "Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing",
      month = apr,
      year = "2021",
      address = "Kiyv, Ukraine",
      publisher = "Association for Computational Linguistics",
      url = "https://www.aclweb.org/anthology/2021.bsnlp-1.1",
      pages = "1--10",
  }
  ```
  
  ## Authors
  The model was trained by [**Machine Learning Research Team at Allegro**](https://ml.allegro.tech/) and [**Linguistic Engineering Group at Institute of Computer Science, Polish Academy of Sciences**](http://zil.ipipan.waw.pl/).
  
  You can contact us at: <a href="mailto:klejbenchmark@allegro.pl">klejbenchmark@allegro.pl</a>