The overhead of the for loop itself is already smaller than you think it is (at least in my experience it usually is the case that people overestimate the impact of the Python parts), but the JIT will use some unrolling to reduce it further. Two key optimizations done by the JIT are to fuse the pointwise computations in RNNs and to combine certain matrix multiplications into one.
Packed sequences are mostly a CuDNN thing, and I guess PyTorch got it from there.
For unidirectional RNNs, just keeping track of the lengths somewhere and letting the RNN run over the padding might be reasonably performant (that won’t work for bi-directional RNNs, though).
I think making RNNs with masks work with good performance in the JIT would be quite doable, but I’m not immediately aware of examples.
Best regards
Thomas