TANKUN_LI:

Do you think that may be there are too many separated networks that slow down the communication speed between GPUs?


This might be the case, as you would have to scatter and gather the results for each model.
A quick check would be to wrap all models in an nn.Sequential container and pass this to nn.DataParallel.
We generally recommend to use DistributedDataParallel with a single process per device, as it would give you the best speedup.