Since GPT-J requires at least 24GB to load the model, you definitely need to use model-parallelism, Therefore I can suggest to you that you take a look at notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub 11
I am going to create an example after have the new DLCs out to show how someone can fine-tune GPT-J with model parallelism.