Hey @meilu_zhu
Sorry about the delay. The grad averaging algorithm is implemented in the reducer 2. Each DistributedDataParallel creates its reducer instance in the constructor 1. More specifically, allreduce is invoked here 1.