Hi, the distributed package assumes that the user will use one process per GPU, so you’d want to use 8 processes in this case. Is there any reason you’re trying to use 12 processes for 8 GPUs? The underlying root cause is that libraries like NCCL won’t work well if multiple processes try to use the same GPU, resulting in deadlocks, hangs etc.