this is normal.
The default implementations are actually highly optimized in C++ and CUDA.
Your implementation involves a dynamic for loop that can not be optimized and therefore is much slower.
You can check the new blog post about custom RNN implementations here 7