When I replace the feature encoder layers of my semantic segmentation models with pretrained VGG16 from torchvision I always encounter that python runs out of cuda memory (12GB).
I wonder how this can be when the models should be equal (I have no problems with cuda when hardcoding the complete network definition myself).
Can it be that pytorch does not free up the memory from unused layers of model.vgg16()?
Source code can be found here 25.