if you want to store the previous outputs, you can use a simple list and do append() and pop().
Apart from this, to your question, the computation graph will be present in the memory until you call backward() on terminal nodes. During back propagation, the computation graph will be destroyed. If you want to keep the computation graph, use loss.backward(retain_graph=True).