Because it depends in which order you do the op. As the high level rnn does not check that directly but its constituent functions do.
And in these functions, the first arg is usually the input given by the user. And we expect everything to be of the same time as it. Hence the error you see.