Thanks @mrshenli! @887574002, there’s not much in your script you’d have to do to be “TorchElastic compliant” take a look at http://pytorch.org/elastic/0.2.0/train_script.html for the details. In order to minimize work lost, you’d have to checkpoint at an interval and ensure that your train script can load from the “most recent checkpoint”.
Aside from this, you’d have to setup an etcd server which is usually trivial for most users but depends on your specific runtime environment.