Can you set the env var CUDA_VISIBLE_DEVICES and specify which GPUs to use before launching training? This will at least clarify whether this is a problem with processes binding to GPUs or something else.
Here’s a good explanation of the env var: cuda - How do I select which GPU to run a job on? - Stack Overflow