Hello I have a problem to think about LSTM. I have a video classification task problem. In the most of people works, they will use CNN to extract the feature, let say for example VGG16. The extracted features will be feed forward through LSTM layer and say that we need to concern about 10 frames, so it will output 1 prediction after 10 frames. So the network will works as CNN extract 10 frames(should be 4096 x 10), concate the feature and feed into LSTM. If we follow this scenario, we must train CNN separately first, but most of people will using pretrained model. Then we train LSTM using the extracted feature before. This will leads into problem that I can not beckpropagate to my previous CNN during my LSTM training.
My question is how can I do in pytorch where I put CNN followed by LSTM, and during the training of LSTM I also can change the weight/ backprop to my CNN?