If you want to backpropagate through the whole thing then pytorch needs to remember a lot of details for the backward pass. I assume that pytorch is pretty good at saving only what it needs to, so you probably wont be able to optimise that much.
On the other hand, if you are using pre-trained weights for most of the vgg layers then you should be able to tell pytorch that those layers don’t need backpropagating.
Something like this ought to work.
# tell pytorch not to optimise those layers
for param in params_of_layers_that_dont_need_training:
    param.requires_grad = False

# do forward like this
with torch.no_grad():
    layers_that_dont_need_training()
layers_that_do_need_training()