Dear @blackberry  Many thanks for your answer!
I see, we are “artificially changing the batch size” when doing multiple forward propagations.
I understand it would be better to stick to known behaviors, but I find it interesting to do multiple forward propagations because it allows for more flexibility of the network.
So referring to case args.GRAD == True of my previous example. And referring to the figure below representing one single learnable weight W0. How will W0 be updated after the backpropagation?
Steps.
1 forward propagate inputs (will create G0 to populate the gradient)
2 forward propagate other inputs (will create G0 to populate the gradient), creates new dynamic graph linked to the same network and rooted at LOSS
3 Calculate backward (green dashed arrows). Now we have two gradient values assigned to the same weight
4 How is W0 Updated? What’s the mechanism?
Could you give me some more hints? Maybe a toy example to figure out what is going on?
@ptrblck @albanD @blackberry
FWD_grad572×546 25.1 KB