sorry for bringing up this old question again, but I’ve been wondering over the same problem for a while.
I understand that once the backProp is done on a module, the gradients of non-leaf parameters will be freed (by checking .grad attribute). But what about intermediate outputs in forwardProp? Will they be freed at the same time?
Thanks.