I would love to be involved with this!
Yeah, we could GPT-J, but we might need to choose smaller model (maybe 1B ?) due to the time and compute constraint.
The HF PR is for PT, adding JAX version will take some time. But it’s somewhat similar to GPT2 which is already available in flax, which we could use.
I wrote a JAX version of VQGAN 8 which can be used as the image tokenizer.