Hey @Tyan
The figure you shared looks a little different from the one @karan_purohit attached. Looks like all processes step into cuda:0, which could happen if they use cuda:0 as the default device and then some tensors/context were unintentionally created there. E.g., when you call empty_cache() without a device context, or create some cuda tensor without specifying device affinity.
Can you try setting CUDA_VISIBLE_DEVICES for all processes so that each process exclusively works on one device?