Are you storing the loss or some other variable, which is attached to the computation graph, somewhere?
Make sure to call detach on your model output etc., if you would like to store the predictions, loss etc. for further analysis.