How did you set CUDA_VISIBLE_DEVICES?  Did you set it in the very beginning of every spawned subprocess?
One way to verify is to add the following line right before where init_process_group is called in each subprocess.
print(os.environ['CUDA_VISIBLE_DEVICES'])