Given the setup you described, I believe all you need to do is to load a pre-trained VGG network using torchvision.models.vgg 25 and feed the output features to the RNN (LSTM, etc.).  Autograd 23 will take care of back-propagating the error through the VGG network. Unless I’m completely missing something, it’s really as simple as that.