The problem was caused by the invocation model(seq, (h, c)), by not passing the (h, c) tensors (they will be automatically initialised internally by PyTorch, I assume), the LSTM produces the output of the correct size.