Hello @danurahul,
Thanks for opening the thread. EleutherAI/gpt-j-6B is not yet trainable with Amazon SageMaker, since the PR is not yet merged into transformers for GPT-J and when it is merged, we need to update the DLC or you have to include the new version of transformers in the requirements.txt.
In addition to this is GPT-J-6B 22GB big and won’t fit on a singe ml.p3.2xlarge instance it would be possible to train it, when merged with distributed training, see Run training on Amazon SageMaker 9
Also even adjusting the two mentioned points above your script would still not work since you are missing a few hyperparameters. The most crucial on is train_file which should have your input file as value, in your case it would be:
/opt/ml/input/data/training/domain-gen-training.jsonl