I’m reading the code about DataParallel but I don’t quite understand why module replicate happens during call to forward.
I thought we should replicate module once and execute forward for multiple times? Or is this class design for training phase because the backward prop would require parameter averaging (synchronization) after each forward call?

Code: pytorch/data_parallel.py at 5b4c3a9da11120f60d732af505bc65f79df14637 · pytorch/pytorch · GitHub