Thank you for your response Tom!
Just to be clear, psuedo-syntactically this would be passing my feature vector to the lstm classifier, finding the class score before the logsoftmax, backwarding this score to the network(which is a 53x15 dim matrix) using .backward(score).
Then using this, I would find the maximally activating points in the feature vector, perhaps use this as a binary encoded mask and backward each of those vectors (since a single vector represents an image) through my CNN for all images(vectors in the 53 length sequence)
Is that right ?