The output of nn.Conv2d is of shape batch_size x num_channels x height x width, how do you want it to be interpretted t as a sequence? nn.RNN() takes a sequence_length x batch_size x input_size shape tensor as input. I think the error comes from the mismatching between the output of nn.Conv2d() and the input of nn.RNN().