Hey there, thanks for the reply.
Which tensors would you store as Variables? I’ve tried deleting the forward returns right after but that didn’t help.
I’ve also noticed something that may be of interest: when watching nvidia-smi I can see that during training the usage goes up to maximum and then back down and does that in cycles whereas during validation the usage is steady at 1/4 of the available gpu ram.
The whole thing that I am trying to develop is a bit complicated but it boils down to: given an image X, have the model predict boundary boxes, use those boxes to extract patches and then recursively do that a number of times. Each patch goes through the same CNN and the extracted features are collected into a list. In the end all the extracted features are concatenated and a fcnn is used for classification.
A colleague suggested http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-python-memory-leaks/ 5 but i am not sure whether it will turn out to be useful. What do you think?