Solved by philschmid in post #6 
              
              
                You could use checkpointing with sagemaker, which automatically syncs the checkpoints to S3 when they are created and only save the model without checkpoints at the end to /opt/ml/model