Solved by mrshenli in post #2 
              
              
                Is it possible that other ranks continuing training next epoch without waiting for rank_0 to finish evaluation?


Yes, it is possible. Because all communication/synchronization happen in the backward pass. So other ranks will proceed with their next forward pass and local backward pass, and then b…