Is there such a thing as an order-$k$ RNN? In HMMs it's common to include edges from previous layers of the model. Looking back at the previous $k$ layers creates an order-$k$ Markov model.
Is this commonly done in RNNs? Have you ever seen this? What was the advantage for that specific application?
In probabilistic graphical models like HMMs, order-$k$ models can be reduced to order-1 models on transformed variables with higher-dimensional support (i.e., the tree decomposition of the order-$k$ graph). Is there a corresponding equivalence for order-$k$ RNNs, where the RNN accepts edges only from the previous layer, but where each layer has greater complexity?

Turning my comment into an answer:

This is doable, but it's not common.
The assumption in an RNN is that the hidden state already carries with it information from all previous states. This isn't true in (e.g.) a $k$th-order HMM. By the Markov assumption, you absolutely can only look back $k$ steps.
I don't think the higher-order RNN would have much value, unless you're trying to include a specific inductive bias of dependency on specifically $k$ steps back. What you've provided is another avenue for information to flow, but unlike the HMM there was already one avenue.
