What is the hidden size and sequence length of your RNN? We’ve seen a 10x gap before, but only for very small hidden size and very long sequences. Since CUDNN can’t run custom cells, what Theano is most likely doing is using its graph optimizer to fuse multiple pointwise CUDA kernels into one. PyTorch currently has inefficient kernel launches because of excess getDevice calls (https://github.com/pytorch/pytorch/issues/917 32) so it should be less bad after those are fixed; eventually we’ll want to provide a way to create custom fused pointwise kernels for things like user-defined RNN cells, at which point it’ll be as fast as Theano for your use case.