Hello there,
Currently I’ve been working on my thesis for video classification, I use CNN as feature extractor and use the output from cnn (-1) as input on RNN.
This is my code for RNN
num_classes = 2
input_size = 2048
hidden_size = 128
batch_size = 1
num_layers = 2 

class Model(nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.rnn = nn.GRU(input_size=input_size, num_layers=num_layers, hidden_size=hidden_size, batch_first=True, dropout=1)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, embedded, seq_len):
        hidden = self.init_hidden(seq_len)

		# Pack them up nicely
        embedded = embedded.view(seq_len, batch_size, input_size)
        # propagate input through RNN
        out, hidden = self.rnn(embedded, hidden)
        out = self.fc(out[-1])
        if use_gpu:
            out = out.cuda()

        return out

    def init_hidden(self, seq_len):

        if use_gpu:
            return Variable(torch.zeros(num_layers, seq_len, hidden_size), requires_grad=True).cuda()

        return Variable(torch.zeros(num_layers, seq_len, hidden_size), requires_grad=True)

The code run well but I got ugly result.
On the train data it successfully to predict on two category, however it predicts only last category on test data. Here’s the log on the training process
train Loss: 0.6007287226361679 Acc: 0.7712418300653595 category1 corrects: 157 category2 corrects: 79
test Loss: 1.1236150416044088 Acc: 0.5384615384615384 category1 corrects: 0 category2 corrects: 35
train Loss: 0.7298363817283531 Acc: 0.6209150326797386 category1 corrects: 94 category2 corrects: 96
test Loss: 0.8569914045242163 Acc: 0.5384615384615384 category1 corrects: 0 category2 corrects: 35
train Loss: 0.7280107533522681 Acc: 0.5751633986928104 category1 corrects: 83 category2 corrects: 93
test Loss: 0.765527761899508 Acc: 0.5384615384615384 category1 corrects: 0 category2 corrects: 35
train Loss: 0.7309918669508952 Acc: 0.5228758169934641 category1 corrects: 73 category2 corrects: 87
test Loss: 0.7198472220164079 Acc: 0.5384615384615384 category1 corrects: 0 category2 corrects: 35
train Loss: 0.723503941901369 Acc: 0.48366013071895425 category1 corrects: 69 category2 corrects: 79
test Loss: 0.7001942808811481 Acc: 0.5384615384615384 category1 corrects: 0 category2 corrects: 35

I dont know why my rnn never predict well on category1 (corrects always 0).
My dataset is likely small, 150 for training and 35 for testing.
I would be happy if someone here help me solve this issue 
Thanks.