Could you please share the cmd you used to launch the processes?
Does the problem disappear if you set CUDA_VISIBLE_DEVICES when launching the process and not passing in --gpu (let it use the default and only visible one)?