Do you reload the model etc. in the val_loader loop on purpose?

Yep. I wouldn’t run code like this in practice, but I just wanted to demonstrate that I’m aggressively deleting/deallocating memory in python’s eyes.

Could you move it in front of the loop and check again, if the memory is increasing?

This is the way I typically run things, but to sanity check, I verified that the memory is still increasing (each minibatch is just processed more quickly since there’s less in-loop overhead)

Could you check that or give some info on the implementation of these methods?

Sure. You can take a look at the whole github repository 3, but I can quickly give an overview:


plf.PartialXentropy(...) is an object that has attributes that points to the neural net adv_trained_net, as well as another implicitly defined neural net

aa.LInfPGD is also an object that points to the neural net adv_trained_net as well as the plf.PartialXentropy(...) object

aa.LInfPGD(...).attack(...) iteratively updates a Variable that is initalized to have batch.data by performing gradient ascent on the loss with respect to the inputs.

So I guess my understanding was that as long as python doesn’t have a reference to an object and I call try to clear the cuda cache, then any pytorch-initialized objects should be deallocated, but this line:

Thus this memory might be collected, since PyTorch cannot free it.

Suggests that maybe that’s not the case?
I’ll try to find a cleaner example to recreate that doesn’t rely on so much application-specific code.