From @Chris’ Stackoverflow post:
I’m by no means a PyTorch expert, but that snippet looks fishy to me:
    # Put the embedded inputs into the GRU.
    output, hidden = self.gru(embedded, hidden)
    # Matrix manipulation magic.
    batch_size, sequence_len, hidden_size = output.shape
    # Technically, linear layer takes a 2-D matrix as input, so more manipulation...
    output = output.contiguous().view(batch_size * sequence_len, hidden_size)


When GRU is not instantiated with batch_first=True, then the output shape is (seq_len, batch, num_directions * hidden_size) – not that seq_len and batch_size are flipped. For the view command it actually doesn’t technically matter, but that’s my main issue here.

view(batch_size * sequence_len, hidden_size) doesn’t look right at all. Say you start with a batch of size 32, but after that you have size of 32*seq_len. Usually, only the output of the last step is used (or the average or the max over all steps)

Something like this should work:
    # Put the embedded inputs into the GRU.
    output, hidden = self.gru(embedded, hidden)
    # Not needed, just to show the true output shape order
    seq_len, batch_size, hidden_size = output.shape
    # Given the shape of output, this is the last step
    output = output[-1]
    # output.shape = (batch_size, hidden_size) <-- What you want

Two personal words of warning:


view() is a dangerous command! PyTorch or any other framework only throws errors when the dimensions of the tensors do not match up. But just because the dimensions fit after view() does not mean the reshaping was done correctly, i.e., that the values are in the right spot of the output tensor. For example, if you have to flatten a shape (seq_len, batch_size, hidden_size) to (batch_size, seq_len*hidden_size), you cannot simply do view(batch_size, -1), but first have to do transpose(1,0) to get a shape of (batch_size, seq_len, hidden_size). With out without transpose(), view() will work and the dimensions will be correct. But only with transpose(), the values are at the right position after view()

Since this is such an easy mistake to make, I saw many examples on GitHub and such where in my opinion it’s no done correctly. The problem is that the network often still learns something. In short, I’m not much more careful when looking and adopting code snippets and the view() command is in my opinion the biggest trap.

If it helps, here’s the forward method of a GRU classifier network:
def forward(self, batch, method='last_step'):
    embeds = self.word_embeddings(batch)
    x = torch.transpose(embeds, 0, 1)
    x, self.hidden = self.gru(x, self.hidden)

    if method == 'last_step':
        x = x[-1]
    elif method == 'average_pooling':
        x = torch.sum(x, dim=0) / len(batch[0])
    elif method == 'max_pooling':
        x, _ = torch.max(x, dim=0)
    else:
        raise Exception('Unknown method.')
    # A series of Linear layers with ReLU and Dropout
    for l in self.linears:
        x = l(x)
    log_probs = F.log_softmax(x, dim=1)
    return log_probs