Sounds good, one question though - So during training for example for a 53 length sequence feature matrix my ground truth was 53 length long.
During testing when I pass a 53 length sequence, the output of the network wont be the same class throughout, it may be that the first 16 outputs for the sequence was of class ‘A’ and the next 37 were of class ‘B’, for my final result I take the majority vote of these predictions.
So in the case of taking gradients, conceptually what would be done ? Should I take the derrivative of the output[class_A_idx] wrt the input, slice it so that I only have the first 16 feature vectors and concatenate with derrivative of  the rest ?
The whole reason why I want a such a saliency map is that I want to see if the transient ‘heat map’ of the first 16 frames would gradually converge to the ‘steady state heat map’ of a training video belonging to class B