If I understand you correctly, you try to train a LSTM network on extracted features using VGG. And you only want to train the LSTM model, not the VGG one.
Why don’t you extract the features of all the frames using VGG and store them in a file. And then train the LSTM model on the extracted features. This would not only be more memory efficient, but also much faster since you can skip the forward pass of the VGG model for training the LSTM model.