Solved by googlebot in post #2 
              
              
                cat/stack is pretty fast, you’re likely measuring asynchronous cuda calls incorrectly. Try the built-in profiler. RNNs are optimized (C++), yes. JIT may help you somewhat, but in general recurrent stuff is not fast.