yes good points 



 pierreguillou:

For example, I’m fine-tuning a T5 large with a dataset of millions of data. I can guarantee that all my training work will last more than 1 day (2 or 3 I think)


The max duration of a SageMaker Training job is 5 days. So you could set this as max_run if you want to run for as long as possible. If you want to run something longer, you can try chaining together multiple checkpointed training jobs (I haven’t seen it done yet though), for example using a Lambda launching job N+1 when triggered by the notification emitted when job N stops



 pierreguillou:

If I can’t calculate the whole training time but I have to setup a max_run , it means I will kill my whole training job before its end… and clearly, I do not want to do that.


you can take a big enough max_run. (less than 5 days though) Or if you checkpoint your training state, you can restart it later from another training job, so that interruptions don’t really matter.
Also note that if you train things for multiple days on one p3.2xlarge and want to go faster, it’s worth considering data-parallel training (example here Distributed Training: Train BART/T5 for Summarization using 🤗 Transformers and Amazon SageMaker)