anandaltekar:

Or do I have to reshape it manually?
I also want to implement Teacher Forcing so I will be modifying the RNN class


You can’t pass input image size of (3 , 128 , 128) to LSTM. You should reshape to (batch,seq,feature). For example input image size of (3128128) -> (1,128,3 * 128) or (1,3,128 * 128) . I think you need the CNN to extract feature before pass into LSTM.