i found that only use the rank 0 model trained with distributeddataparallel to inference on val dataset , performance is not as good as use all the model  trained with distributeddataparallel to inference on val dataset, only use rank 0 model usually get 0.5% to 1% accuracy slower in classification task. so do we need to allreduce the model in the training , for example, we can allreduce the model begin or after every epoch? hope for your detailed kind reply . Thanks very much.