Hi -
I have images in sequences of variable length. I am trying to first process each image with a CNN to get a feature representation. Once I have variable-length sequences of features, I will process each sequence through an LSTM. I know that I can pad the variable-length sequence of feature vectors with zeros and create a packed sequence with pack_padded_sequence() before sending it to LSTM. But how can I do the same to my data before sending it to CNN? I do not want to pad the image sequence with zeros and let the CNN waste computation on processing images of all zeros. Any thoughts?
The way I am currently hacking it is that: I am still padding the image sequence. Once I get a PackedSequence object using pack_padded_sequence(), I can call the ‘forward function’ of my CNN model on the ‘data’ attribute of the PackedSequence object. I can then manually construct a new PackedSequence object with the output from my CNN model, effectively replacing the data field of the old PackedSequence object. Finally, I send the new PackedSequence to the downstream LSTM. Is this an okay workaround?
Thanks!