The code you have doesn’t use GPUs, it’s CPU only tensors.
I suggest add some print statements in your model before and after critical sections of the code, i.e.
dist.init_process_group(“gloo”, rank=rank, world_size=world_size)
forward pass
backward pass
optimizer.step()
and see which line causes your error.