Solved by mrshenli in post #2 
              
              
                Hey @aguirguis, if you are looking for elastic training for distributed data parallel, torchelastic is built for this purpose. It will conduct re-rendezvous on living nodes when failure occurs.