Since you are training on multiple nodes of a cluster, what does your output of nvidia-smi look like? I think you shouldn’t use CUDA_VISIBLE_DEVICES=0,1,2 as not all 3 GPUs are bot connected and visible to the same system.
CUDA_VISIBLE_DEVICES is used only when there are multiple GPUs in the system and we want to run on selected GPUs. Hope this solves some part of it.
Can you put the link for the tutorial for more help? Thanks.