Hi,
I want to use data parallel to train my model on single GPU. I followed the example of Pytorch DISTRIBUTED DATA PARALLEL, and pass the same device_id to 4 processes. With all-reduce sync method, it runs even slower than using a single process. The interesting thing is by disabling all_reduce sync-up for gradients, there is a great speed up of training itself. So I think the GPU has extra compute capacity for multiple training process, but the bottleneck is all-reduce method. Any one know the reason for this bottleneck? Is there any other way to sync param gradients without using all_reduce? Thanks.