Overview
The GPT-J model was released in the kingoflolz/mesh-transformer-jax repository by Ben Wang and Aran Komatsuzaki. It is a GPT-2-like
causal language model trained on the Pile dataset.
This model was contributed by Stella Biderman.
Tips:
To load GPT-J in float32 one would need at least 2x model size CPU
  RAM: 1x for initial weights and another 1x to load the checkpoint. So for GPT-J it would take at least 48GB of CPU
  RAM to just load the model. To reduce the CPU RAM usage there are a few options. The torch_dtype argument can be
  used to initialize the model in half-precision. And the low_cpu_mem_usage argument can be used to keep the RAM
  usage to 1x. There is also a fp16 branch which stores
  the fp16 weights, which could be used to further minimize the RAM usage. Combining all this it should take roughly
