This sounds like a more or less finished team already - cool  Let’s note it down on Monday 
I like the idea and think it’d be cool to make this project happen!
GPT-J and GPT2 are very similar, so it might be easiest to just use the FlaxGPT2 implementation actually  If you think you require model parallelism however, the official GPT-J repo could be the way to go 
In any way the run_flax_clm.py: transformers/run_clm_flax.py at master · huggingface/transformers · GitHub 4 should be helpful here 
Also for the massive amount of data, maybe dataset streaming could help? Load a Dataset in Streaming mode — datasets 1.8.0 documentation