Are you seeing the increase in memory, if you push another model to the device or do you delete the first one?
You approach looks valid. Alternatively you could try to replace the fc7 layer with an nn.Identity() module.