I have not used DistributedDataParallel yet, so I am not sure about the right way to use it.
I would suggest that you first write code that works on a single GPU, just to ensure that it works properly without the parallelization. You could then try using DataParallel (which involves adding just one extra line of code) to see if that speeds it up sufficiently for your needs. Maybe it will, and then you don’t have to worry about the more involved DistributedDataParallel.