readme: ---
  language: pl
  thumbnail: https://raw.githubusercontent.com/kldarek/polbert/master/img/polbert.png
  ---
  
  # Polbert - Polish BERT
  Polish version of BERT language model is here! It is now available in two variants: cased and uncased, both can be downloaded and used via HuggingFace transformers library. I recommend using the cased model, more info on the differences and benchmark results below. 
  
  ![PolBERT image](https://raw.githubusercontent.com/kldarek/polbert/master/img/polbert.png)
  
  ## Cased and uncased variants
  
  * I initially trained the uncased model, the corpus and training details are referenced below. Here are some issues I found after I published the uncased model:
      * Some Polish characters and accents are not tokenized correctly through the BERT tokenizer when applying lowercase. This doesn't impact sequence classification much, but may influence token classfication tasks significantly.
      * I noticed a lot of duplicates in the Open Subtitles dataset, which dominates the training corpus.
      * I didn't use Whole Word Masking. 
  * The cased model improves on the uncased model in the following ways:
      * All Polish characters and accents should now be tokenized correctly. 
      * I removed duplicates from Open Subtitles dataset. The corpus is smaller, but more balanced now. 
      * The model is trained with Whole Word Masking. 
  
  ## Pre-training corpora
  
  Below is the list of corpora used along with the output of `wc` command (counting lines, words and characters). These corpora were divided into sentences with srxsegmenter (see references), concatenated and tokenized with HuggingFace BERT Tokenizer. 
  
  ### Uncased
  
  | Tables        | Lines           | Words  | Characters  |
  | ------------- |--------------:| -----:| -----:|
  | [Polish subset of Open Subtitles](http://opus.nlpl.eu/OpenSubtitles-v2018.php)      | 236635408| 1431199601 | 7628097730 |
  | [Polish subset of ParaCrawl](http://opus.nlpl.eu/ParaCrawl.php)     | 8470950      |   176670885 | 1163505275 |
  | [Polish Parliamentary Corpus](http://clip.ipipan.waw.pl/PPC) | 9799859      |    121154785 | 938896963 |
  | [Polish Wikipedia - Feb 2020](https://dumps.wikimedia.org/plwiki/latest/plwiki-latest-pages-articles.xml.bz2) | 8014206      |    132067986 | 1015849191 |
  | Total | 262920423      |    1861093257 | 10746349159 |
  
  ### Cased
  
  | Tables        | Lines           | Words  | Characters  |
  | ------------- |--------------:| -----:| -----:|
  | [Polish subset of Open Subtitles (Deduplicated) ](http://opus.nlpl.eu/OpenSubtitles-v2018.php)      | 41998942| 213590656 | 1424873235 |
  | [Polish subset of ParaCrawl](http://opus.nlpl.eu/ParaCrawl.php)     | 8470950      |   176670885 | 1163505275 |
  | [Polish Parliamentary Corpus](http://clip.ipipan.waw.pl/PPC) | 9799859      |    121154785 | 938896963 |
  | [Polish Wikipedia - Feb 2020](https://dumps.wikimedia.org/plwiki/latest/plwiki-latest-pages-articles.xml.bz2) | 8014206      |    132067986 | 1015849191 |
  | Total | 68283960      |    646479197 | 4543124667 |
  
  
  ## Pre-training details
  
  ### Uncased 
  
  * Polbert was trained with code provided in Google BERT's github repository (https://github.com/google-research/bert)
  * Currently released model follows bert-base-uncased model architecture (12-layer, 768-hidden, 12-heads, 110M parameters)
  * Training set-up: in total 1 million training steps: 
      * 100.000 steps - 128 sequence length, batch size 512, learning rate 1e-4 (10.000 steps warmup)
      * 800.000 steps - 128 sequence length, batch size 512, learning rate 5e-5
      * 100.000 steps - 512 sequence length, batch size 256, learning rate 2e-5
  * The model was trained on a single Google Cloud TPU v3-8 
  
  ### Cased
  
  * Same approach as uncased model, with the following differences:
      * Whole Word Masking
  * Training set-up:
      * 100.000 steps - 128 sequence length, batch size 2048, learning rate 1e-4 (10.000 steps warmup)
      * 100.000 steps - 128 sequence length, batch size 2048, learning rate 5e-5
      * 100.000 steps - 512 sequence length, batch size 256, learning rate 2e-5
  
  
  ## Usage
  Polbert is released via [HuggingFace Transformers library](https://huggingface.co/transformers/).
  
  For an example use as language model, see [this notebook](/LM_testing.ipynb) file. 
  
  ### Uncased
  
  ```python
  from transformers import *
  model = BertForMaskedLM.from_pretrained("dkleczek/bert-base-polish-uncased-v1")
  tokenizer = BertTokenizer.from_pretrained("dkleczek/bert-base-polish-uncased-v1")
  nlp = pipeline('fill-mask', model=model, tokenizer=tokenizer)
  for pred in nlp(f"Adam Mickiewicz wielkim polskim {nlp.tokenizer.mask_token} był."):
    print(pred)
  # Output:
  # {'sequence': '[CLS] adam mickiewicz wielkim polskim poeta był. [SEP]', 'score': 0.47196975350379944, 'token': 26596}
  # {'sequence': '[CLS] adam mickiewicz wielkim polskim bohaterem był. [SEP]', 'score': 0.09127858281135559, 'token': 10953}
  # {'sequence': '[CLS] adam mickiewicz wielkim polskim człowiekiem był. [SEP]', 'score': 0.0647173821926117, 'token': 5182}
  # {'sequence': 