Have you tried running a profiler (like nvprof) to see if there is anything preventing the GPUs from making forward progress? This would show you if there is any imbalance between the work the GPUs perform.