Hi Pytorch Community,
I am trying a CNN + LSTM model, using CNN for feature extraction and then passing sequences of the features to LSTM for classification, CNN is trained separately and just used as a bottleneck layer.
I am mainly trying to find how the class activation heat map changes as frames are passed to an LSTM.
So if I have a 53x1024 matrix where 53 is the sequence length, how do I find which part of this matrix or which part of each of the vectors in the 53 length sequence activated the LSTM + MLP part (lstm has 1000 hidden dim and MLP part is nn.Linear(1000,15) where 15 is the number of classes)
After I find this I thought maybe I’ll do visual back prop through the CNN using these most activated parts as a binary map.