Hey @ujjirox,
Could you upload the logs as files maybe? When running distributed training it sometimes happen that the real error is way above the exit.
Without seeing the full error. It might be possible that your batch_size is too big. When scaling up from p3.2xlarge to p3.16xlarge (same GPUs) SageMaker might use more of the GPU memory for the distribution.