Looks like all processes create some states on cuda:5. It’s possible that some part of the code is using default device. One option is to use set_device 1 to set the current device before creating any context/data on the device. Another option is to set CUDA_VISIBLE_DEVICES to make sure that each process can only see one device.
BTW, could you please add a “distributed” tag for distributed-training related questions? Developers working on the distributed package actively monitors that tag.