Here are more details:

Arabic GPT-J-6B
Train a GPT-J-6B on Arabic dataset (Arabic Wikipedia for exmple) using Jax on TPUs.

2. Language
The model will be trained in Arabic.

3. Model
GPT-J-6B

4. Datasets
Arabic Wikipedia.
(TODO: more to come)

5. Training scripts
TODO: Update

6. Challenges
No prior experience doing this kind of work, so this will be mostly a discovery.

7. Desired project outcome
An Arabic GPT-J model that is available on HuggingFace.

8. Reads

Good blog post about GPT-J-6B 24