PreTrain GPT-2 from scratch for German on novel GC4 dataset
To the best of our knowledge, there is no GPT-J model trained on a german corpus.

2. Language
The model will be trained in German.

3. Model
GPT-J

4. Datasets
The German colossal, cleaned Common Crawl corpus. GC4 Corpus — German NLP Group documentation 5
Another potential source of data is that which was used by the Digitale Bibliothek/Münchener Digitalisierungszentrum to pretrain their German GPT-2 variant

5. Training scripts

  
      

      GitHub
  

  
    

kingoflolz/mesh-transformer-jax 6

  Model parallel transformers in JAX and Haiku. Contribute to kingoflolz/mesh-transformer-jax development by creating an account on GitHub.


  

  
    
    
  

  



6. Challenges
Ridiculous amount of data renders both training and preprocessing challenging.

7. Desired project outcome
A competitive German language model, and a demo based on the HF SvelteKit inference demos

8. Team members
Christopher Akiki (myself) and Alina Mailach (@mailach)