Thanks for the pointer. I think that my problem might be a little different.
I have two networks A and B, the output of A will be fed to network B, and the loss is defined on the top of network B. The network B is fixed and network A is to be optimized.
So I need network B to back propagate the gradients from loss to A, but these gradients do not need to be saved since network B doesn’t need to be updated. I am wondering that in this case, can I set all variables in B to be volatile=True?
Is there some way for me to know whether the gradients of parameters in B are buffered or not?