I am not using nn.DataParallel. I am just running 3 different independent programs all on different GPUs that access the same data. In this case, shall I use DistributedDataParallel with a single process for each device. If so, can you please tell me how to use that? Also, would it be better to create 3 copies of data and all 3 process their copy of data? Can this reduct the overhead?
P.S. - I am using AWS’s p3.8xlarge machine
Thanks!