Hi,
Interesting. The main problem with this is that many operations like conv/linear actually do the accumulation during their backward pass: the matrix multiplication that is done during the backward of a linear is doing the accumulation. This means that you would need to rewrite these Modules to handle backward without accumulation. Or forward from a batch of weights (which they don’t at the moment).