class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.gru = nn.GRU(2048, hidden_size)

    def forward(self, input):
        # input should be (num_frames, batch_size, 2048)
        output, hidden = self.gru(input)
        return output, hidden

Should work.
Notes:


you shouldn’t have to manually initialize the hidden state in the encoder rnn as it processes the whole sequence into an encoded representation at one go so you never have to manually pass the state back in.


you  may want to throw those 2048 dim features through a few linear layers first as 2048 dim is pretty large for an rnn.