One way might be printing out the forward output on all rank every N iterations, and see if the loss is different and resides on different GPUs. You can also check nvidia-smi to see if they indeed keep multiple GPUs busy.