Hi @pierreguillou ; the content of opt/ml/model is accessible only at the end of training (and it will be compressed in a model.tar.gz, which will be enormous if you save dozen/hundreds of transformers checkpoints there). If you want to export models to s3 during training, without interruption, in the same file format as saved locally, you need to save them in /opt/ml/checkpoints, and specify the S3 sync location in your SDK call with the parameter checkpoint_s3_uri. Then use the AWS CLI or boto3 download_file to bring the from S3 back to your notebook.
See here the doc of the checkpoint feature, which is in my opinion one of the best features of SageMaker

Use Checkpoints in Amazon SageMaker - Amazon SageMaker 2
notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub