You could test LSTM with num_layers = 2 like this… EDIT: this won’t work because hidden only contains the last timestep.
lstm = nn.LSTM(.., num_layers=2)
_, (hidden, cell) = lstm(input, (hidden, cell))
# Where hidden is a tuple (hidden_layer1, hidden_layer2)
# and cell is a tuple (cell_layer1, cell_layer2)
out = fc(hidden[0] + hidden[1])