Could you try to run the DDP command on a single node and GPU and check the memory usage?
I guess the code might create unnecessary CUDA contexts on other devices, but since the repository contains a lot of files I haven’t looked through all of them.