Based on a recent issue 30 opened for PyTorch, it is in fact the case currently (v1.1.0) that the module will retain the reduction functions and new ones will be added each time the model is wrapped in DDP