Hi, I am curious why this require_forward_param_sync is set to True in the DistributedDataParallel. After I manually set it to False, the multi-gpus in one node speeds up a lot. Since the gradients of each replica have been synchronized, why do we need to synchronize the parameters in the forward?