pierreguillou:

I get it but it looks a bit handmade, right


Agree! there is opportunity for innovation here. I’m not aware of any tool that can predict the training time and resource consumption of a training script given an arbitrary dataset, an arbitrary model graph and an arbitrary training cluster. But why not tackling it as a regression problem, trained on few previous training jobs. I haven’t seen it done much though (there was an attempt to do it for Apache Spark). Could be a unicorn startup idea