@dariodematties To simplify the investigation here, can you check if simple torch CUDA operations work fine (ex: creating tensors on GPUs and performing some basic ops on them)? In addition to this, testing NCCL collective operations like allreduce independently would be a good idea to see if there is something  related to NCCL here.

model = DDP(model)

In this case, I’m assuming the model is actually on CPU and that’s why its working?