Solved by ptrblck in post #2 
              
              
                I think the main design decision for this behavior is, because the gradient calculation and the update steps are independent from each other. 
E.g. you can easily accumulate gradients with multiple forward/backward passes, if a larger batch doesn’t fit in your memory. 
Also, you might work with mult…