sunshichen:

What is the correct way to do validation when DDP model has syncbatchnorm layer? Should I do validation on all ranks?


Yes, you probably need to do validation on all ranks since SyncBatchNorm has collectives which are expected to run on all ranks. The validation is probably getting stuck since SyncBatchNorm on rank 0 is waiting for collectives from other ranks.
Another option is to convert the SyncBatchNorm layer to a regular BatchNorm layer and then do the validation on a single rank.