Solved by cbalioglu in post #3 
              
              
                In traditional HPC and ML systems all nodes that are part of a job are started simultaneously (as you implied in your question) and the job gets dispatched (by a scheduler) once all nodes are ready to execute. The set of nodes that are part of a job are usually called a “gang”. 
TorchElastic is desi…