they are initially freed during the backward operation for efficiency. You can use loss.backward(retain_graph=True) in order to keep the computation graph, however at this precise moment I am not sure if you can access the gradient of the operation. Maybe you can take a look at the documentation on the autograd package.