Nikhil_Pinnaparaju:

Each validation step pred will be reinitialized so I don’t believe it will cause too big of a problem.


You could store each loss using preds.append(loss.detach()) and see, if the memory growth disappears.



 Nikhil_Pinnaparaju:

Now do I reset this and remove the memory they consume after each forward step.


The backward call will clean the computation graph and its intermediate tensors, if you don’t run it with retain_graph=True.
As long as some tensors hole a reference to the computation graph, they are stored and use memory.