readme: ---
  language:
  - de
  widget:
  - text: "Der Sinn des Lebens ist"
    example_title: "Sinn des Lebens"
  inference:
    parameters:
      temperature: 0.7
      repetition_penalty: 1.4
  ---
  
  # Easy German GPT2 Model
  
  A language model for german easy language ("leichte Sprache") based on [German GPT-2 model](https://huggingface.co/dbmdz/german-gpt2)
  
  ## Model Details
  
  Initialized using the weights of [German GPT-2 model](https://huggingface.co/dbmdz/german-gpt2).  
  Then fine-tuned for one epoch on "leichte Sprache" corpora consisting of:
   - encyclopedia like data
   - news like data
   
   Hyperparamters used for fine-tuning:
   - tokenizer:
     - max_length: 1024 (but trained with dynamic length, using the collator functions 'pad_to_multiple_of=8')
     - stride: 64
     - return_overflowing_tokens=True
     
   - training arguments
     - num_train_epochs=1
     - learning_rate=1e-3
     - weight_decay=0.01
     - per_device_train_batch_size=4 
     - gradient_accumulation_steps=4
     - warmup_steps=200
     - fp16=True
     
   → 25112 training items, trained on google colab GPU (30 min)
     
   
   ## Evaluation results
   
  The perplexity value is calculated based on an unseen dataset containing manually aligned standard german and "leichte Sprache" texts.  
  For calculation, the method described in [this tutorial](https://huggingface.co/docs/transformers/perplexity) was used with the following values:  
  - max_length = 512
  - stride = 256
  
  For comparison: running the modified function on [this example](https://huggingface.co/docs/transformers/perplexity#example-calculating-perplexity-with-gpt2-in-transformers) gives us a perplexity score of **18.2551**
  
  | Model | Perplexity "leichte Sprache" [(Easy MDR News)](https://www.mdr.de/nachrichten-leicht/index.html)) |Perplexity standard german ([Standard MDR News](https://www.mdr.de/nachrichten/index.html))|
  |:-----:|:----------------------------------------:|:------------------------------------:|
  |[German GPT-2 model](https://huggingface.co/dbmdz/german-gpt2)| 23.8257   | **24.0301** |
  |our model| **17.3053**   | 48.6314 |