I think they are freed during the backward call (at least it can be done that way).
Derivatives are basically transposed matrix operations for fully connected and transposed convolutions for convolution operations. So as you go back to the initial layer you can just compute the operation and free the memory. Anyway I am not pretty sure how autograd works but I think that yes, they should be some place in the c-code where the value is momentarily stored, maybe @ptrblck or @albanD now some details on this.