 Replication of [gpt2-wechsel-german](https://huggingface.co/benjamin/gpt2-wechsel-german)  - trained with [BigScience's DeepSpeed-Megatron-LM code base](https://github.com/bigscience-workshop/Megatron-DeepSpeed) - 22hrs on 4xA100 GPUs (~ 80 TFLOPs / GPU) - stopped after 100k steps - less than a single epoch on `oscar_unshuffled_deduplicated_de` (excluding validation set; original model was trained for 75 epochs on less data) - bf16 - zero stage 1 - tp/pp = 1  