OlivierCR:

no you don’t need to do that, this is a maximum boundary that you set.
By default it is one day (86400s).


Thanks but I don’t understand. For example, I’m fine-tuning a T5 large with a dataset of millions of data. I can guarantee that all my training work will last more than 1 day (2 or 3 I think) on a ml.p3.2xlarge instance.
If I can’t calculate the whole training time but I have to setup a max_run, it means I will kill my whole training job before its end… and clearly, I do not want to do that.



 OlivierCR:

how to estimate the duration of a training, just train for a few epochs (or iterations if an epoch is too long), then scale to the desired number of iterations or epochs


Ok … I get it but it looks a bit handmade, right? (when the goal of Hugging face and AWS SageMaker is to help automate the use of transformers)