Thanks for the reply.  I believe I found the error.  The original code was written for a single GPU and unfortunately there were multiple places where cuda:0 was hardcoded.
However, I do have a followup question.  From all the examples I have seen so far, the dataloader is associated with DistributedSampler: https://pytorch.org/docs/stable/_modules/torch/utils/data/distributed.html 12
In many cases the dataloader loads files in a directory.  In the case of using DDP, the DistributedSampler would be used to allocate files for each GPU such that each GPU would get a unique distribution of samples from the total dataset (I am assuming that the total data items is integer divisible by the number of GPUs).
In my case I am loading 3D data and then take patches of this data, which serves as the training input of the network.  So one data file corresponds to more than one training input.
What I do currently is that I load the data by loading the files in a folder.  Then I use a custom sampler that loads the patch indices and has an iterator that passes patches when called.  I feed this sampler to a Dataloader, where I pass in the whole data, the sampler, and batch size.  This works fine for one GPU.
I am now moving on to converting my code to DDP.  I could put the DistributedSampler after my custom sampler, but I worry about the idea of multiple GPUs accessing the same file (again the input is a patch and different patches can come from the same file).  Am I correct to say that this would be a problem?
Another approach could be to put the DistributedSampler before my current sampler.  But I am a bit unsure how to hook up this DistributedSampler to my existing code.
I suppose yet another method could be to bypass using torch.utils.data.distributed.DistributedSampler and perhaps instead have my initial Dataset have a getitem that distributes the files among the GPUs in a manner similar to the DistributedSampler, and then keep the rest of my hooks the same.  Or alternatively, in the main loop I could have the logic for handling the distribution of files and pass this into the spawned process.
Would one approach be better than others?  Or should I be using another approach altogether?  Does the DDP work properly if the code does not use the DistributedSampler?