readme: ---
  language: pl
  ---
  
  # HerBERT tokenizer
  
  **[HerBERT](https://en.wikipedia.org/wiki/Zbigniew_Herbert)** tokenizer is a character level byte-pair encoding with
  vocabulary size of 50k tokens. The tokenizer was trained on [Wolne Lektury](https://wolnelektury.pl/) and a publicly available subset of
  [National Corpus of Polish](http://nkjp.pl/index.php?page=14&lang=0) with [fastBPE](https://github.com/glample/fastBPE) library.
  Tokenizer utilize `XLMTokenizer` implementation from [transformers](https://github.com/huggingface/transformers).
  
  ## Tokenizer usage
  Herbert tokenizer should be used together with [HerBERT model](https://huggingface.co/allegro/herbert-klej-cased-v1):
  ```python
  from transformers import XLMTokenizer, RobertaModel
  
  tokenizer = XLMTokenizer.from_pretrained("allegro/herbert-klej-cased-tokenizer-v1")
  model = RobertaModel.from_pretrained("allegro/herbert-klej-cased-v1")
  
  encoded_input = tokenizer.encode("Kto ma lepszą sztukę, ma lepszy rząd – to jasne.", return_tensors='pt')
  outputs = model(encoded_input)
  ```
  
  ## License
  CC BY-SA 4.0
  
  ## Citation
  If you use this tokenizer, please cite the following paper:
  ```
  @inproceedings{rybak-etal-2020-klej,
      title = "{KLEJ}: Comprehensive Benchmark for {P}olish Language Understanding",
      author = "Rybak, Piotr  and
        Mroczkowski, Robert  and
        Tracz, Janusz  and
        Gawlik, Ireneusz",
      booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
      month = jul,
      year = "2020",
      address = "Online",
      publisher = "Association for Computational Linguistics",
      url = "https://www.aclweb.org/anthology/2020.acl-main.111",
      doi = "10.18653/v1/2020.acl-main.111",
      pages = "1191--1201",
  }
  ```
  
  ## Authors
  Tokenizer was created by **Allegro Machine Learning Research** team.
  
  You can contact us at: <a href="mailto:klejbenchmark@allegro.pl">klejbenchmark@allegro.pl</a>
  