Solved by pritamdamania87 in post #11 
              
              
                @Euruson I think I’ve figured out the problem here. You are still using DDP for the validation phase even though it runs only on one rank. Even though you might not run the backward pass for DDP during eval phase, the forward pass for DDP might still invoke some collective operations (ex: syncing bu…