My guess is that the RNNModel hidden state is detached, repackaged, or reinitialised in between batches, whereas the RecurrentHighwayNetwork’s hidden state is simply kept from one batch to the next, in which case you are backpropagating all the way back to the beginning of the first batch.