Unfortunately I couldn’t reproduce your problem. This is the output of your script when I run it on a 4 GPU machine:
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1025 17:28:00.405782 39617 ProcessGroupNCCL.cpp:520] [Rank 0] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
NCCL_DEBUG: UNSET
I1025 17:28:00.405783 39637 ProcessGroupNCCL.cpp:621] [Rank 0] NCCL watchdog thread started!
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1025 17:28:00.422144 39618 ProcessGroupNCCL.cpp:520] [Rank 1] ProcessGroupNCCL initialized with following options:
NCCL_ASYNC_ERROR_HANDLING: 0
NCCL_BLOCKING_WAIT: 0
TIMEOUT(ms): 1800000
USE_HIGH_PRIORITY_STREAM: 0
NCCL_DEBUG: UNSET
I1025 17:28:00.422158 39638 ProcessGroupNCCL.cpp:621] [Rank 1] NCCL watchdog thread started!
Rank  1  has data  tensor(2., device='cuda:2') tensor(2., device='cuda:3')
Rank  0  has data  tensor(2., device='cuda:0') tensor(2., device='cuda:1')

Are you sure that your script is able to see all your CUDA devices (e.g. CUDA_VISIBLE_DEVICES=0,1,2,3)?