Hidden state entangles input vector elements, so this is impossible I think. You’d have to somehow split your vector before rnn layer, and conditionally combine it back to produce rnn input (residual networks, multi-layer rnns, something like that…)