Huige_Cheng:

RuntimeError: Tensors must be CUDA and dense


Are you using sparse tensors?

Another problem of DIstributedDataParallel is each process using all gpus like below, is this by design ?

How did you construct DDP? You need to either set device_ids 26 arg properly or use the CUDA_VISIBLE_DEVICES env var to configure that, and make sure no DDP processes share the same GPU. Otherwise, each process will try to use all visible devices, and when two DDP process share a GPU, it could hang.