Hello @nreamaroon,
thank you for opening the thread! That is indeed strange.
Did you use your own train.py or already created an example? If you wrote your own one can you share your training script?
Especially, which saving strategy you use. And could you also share the size of the dataset and model you use and the hyperparamter?
The only understandable reason for me currently is that during training on the large dataset SageMaker is creating this many checkpoints that it somehow fails uploading at the end. But this wouldn’t make any sense at all.
An easy way to test this would be only saving the last model in opt/ml/model.
In Addition to this, I have shared your issue with the AWS Team directly!