Some additional links:

Implementation that will be used: GitHub - kingoflolz/mesh-transformer-jax: Model parallel transformers in JAX and Haiku 8

Where the code will live: GitHub - yassineAlouini/arabic-gpt-j-6b: Fine-tuning the GPT-J-6B model on some Arabic datasets using Jax and its ecosystem. 21