Do you need the gradients for your pretrained VGG model or are you using it as a fixed feature extractor?
If the latter is true, you can use Variable(..., volatile=True), so that it will use the absolute minimal amount of memory to evaluate the model.