I experiment with using a for loop to collect every hidden state. The loss doesn’t change, but runtime is ~10x more (with a sequence length of 30).
        embedding = self.drop(self.emb(inputs))
        
        b_len = embedding.size(0)
        outputs = Variable(torch.cuda.FloatTensor(b_len, embedding.size(1), self.hidden_size))
        for i in range(b_len):
            
            cur_emb = embedding[i:i+1, :]#.view(1, embedding.size(1), embedding.size(2))
            o, hidden = self.gru_1(cur_emb, hidden)
            outputs[i, :, :] = o