readme: ---
  language: pl
  tags:
  - text-generation
  widget:
  - text: "Najsmaczniejszy polski owoc to"
  ---
  
  # papuGaPT2 - Polish GPT2 language model
  [GPT2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) was released in 2019 and surprised many with its text generation capability. However, up until very recently, we have not had a strong text generation model in Polish language, which limited the research opportunities for Polish NLP practitioners. With the release of this model, we hope to enable such research. 
  
  Our model follows the standard GPT2 architecture and training approach. We are using a causal language modeling (CLM) objective, which means that the model is trained to predict the next word (token) in a sequence of words (tokens). 
  
  ## Datasets
  We used the Polish subset of the [multilingual Oscar corpus](https://www.aclweb.org/anthology/2020.acl-main.156) to train the model in a self-supervised fashion. 
  
  ```
  from datasets import load_dataset
  dataset = load_dataset('oscar', 'unshuffled_deduplicated_pl')
  ```
  
  ## Intended uses & limitations
  The raw model can be used for text generation or fine-tuned for a downstream task. The model has been trained on data scraped from the web, and can generate text containing intense violence, sexual situations, coarse language and drug use. It also reflects the biases from the dataset (see below for more details). These limitations are likely to transfer to the fine-tuned models as well. At this stage, we do not recommend using the model beyond research.
  
  ## Bias Analysis
  There are many sources of bias embedded in the model and we caution to be mindful of this while exploring the capabilities of this model. We have started a very basic analysis of bias that you can see in [this notebook](https://huggingface.co/flax-community/papuGaPT2/blob/main/papuGaPT2_bias_analysis.ipynb).
  
  ### Gender Bias
  As an example, we generated 50 texts starting with prompts "She/He works as". The image below presents the resulting word clouds of female/male professions. The most salient terms for male professions are: teacher, sales representative, programmer. The most salient terms for female professions are: model, caregiver, receptionist, waitress.
  
  ![gender bias](https://huggingface.co/flax-community/papuGaPT2/raw/main/gender_bias.jpeg)
  
  ### Ethnicity/Nationality/Gender Bias
  We generated 1000 texts to assess bias across ethnicity, nationality and gender vectors. We created prompts with the following scheme: 
  
  * Person - in Polish this is a single word that differentiates both nationality/ethnicity and gender. We assessed the following 5 nationalities/ethnicities: German, Romani, Jewish, Ukrainian, Neutral. The neutral group used generic pronounts ("He/She"). 
  * Topic - we used 5 different topics: 
    * random act: *entered home*
    * said: *said*
    * works as: *works as*
    * intent: Polish *niech* which combined with *he* would roughly translate to *let him ...*
    * define: *is*
  
  Each combination of 5 nationalities x 2 genders x 5 topics had 20 generated texts. 
  
  We used a model trained on [Polish Hate Speech corpus](https://huggingface.co/datasets/hate_speech_pl) to obtain the probability that each generated text contains hate speech. To avoid leakage, we removed the first word identifying the nationality/ethnicity and gender from the generated text before running the hate speech detector.
   
  The following tables and charts demonstrate the intensity of hate speech associated with the generated texts. There is a very clear effect where each of the ethnicities/nationalities score higher than the neutral baseline. 
  
  ![hate score by ethnicity](https://huggingface.co/flax-community/papuGaPT2/raw/main/hate_by_ethnicity.png)
  
  Looking at the gender dimension we see higher hate score associated with males vs. females. 
  
  ![hate score by gender](https://huggingface.co/flax-community/papuGaPT2/raw/main/hate_by_gender.png)
  
  We don't recommend using the GPT2 model beyond research unless a clear mitigation for the biases is provided. 
  
  ## Training procedure
  ### Training scripts
  We used the [causal language modeling script for Flax](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). We would like to thank the authors of that script as it allowed us to complete this training in a very short time!
  
  ### Preprocessing and Training Details
  The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a vocabulary size of 50,257. The inputs are sequences of 512 consecutive tokens.
  
  We have trained the model on a single TPUv3 VM, and due to unforeseen events the training run was split in 3 parts, each time resetting from the final checkpoint with a new optimizer state: 
  1. LR 1e-3, bs 64, linear schedule with warmup for 1000 steps, 10 epochs, stopped after 70,000 steps at eval loss 3.206 and perplexity 24.68
  2. LR 3e-4, bs 64, linear schedule with warmup