Thanks for responding! I use NCCL as backend:
torch.distributed.init_process_group(backend="nccl")

If I change the device to ‘cpu’, there is an error: Tensors must be CUDA and dense
I will try to print them before all_reduce, and see what happens.