Yeah, I know I could just write my own gradient averaging but I lose out on the perf wins. And it seems kind of silly to maintain a fork of DDP just to change the reduction from mean to sum. I’ve looked at the join method on DDP but it doesn’t apply to my issue. Thanks for the reply!