I’m now doubtful of this approach, I set a breakpoint on AccumulateGrad::apply() (used for grad_accumulator_) and it’s just getting called once (per tensor) to call to backward(), so it’s actually a copy rather than an accumulator.