I think the length information is even not required.
Note that we pad the short sequences with zeros. RNN with input hidden=None (the first frame) would actually creates hidden=torch.zeros(...). In this case, [z, y, x, 0, 0] and [0, 0, z, y, x] just produce same results.
But for theoretically, an inverse ([x, y, z, 0, 0] -> [0, 0, z, y, x]) and a shift ([0, 0, z, y, x] -> [z, y, x, 0, 0]) are required in case paddings are not zeros.