Thomas, please, could you check it on your machine? It will take only a few minutes:) Run this tutorialhttp://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
and interrupt training several times after maybe 10 seconds. Will your memory consumption grow?