Yes, this is not great. The outer nn.DataParallel module will replicate N times. The inner modules will also be replicated N times. I think you’ll end up with N^2 replicas instead of just N. You can add some logging to the forward functions to confirm what really ends up happening.