It could be all processes unintentionally created CUDA context on the default GPU (cuda:0). To avoid this situation, can you try setting CUDA_VISIBLE_DEVICES env var to a different device for each process, so that each process would only see one GPU?