You are not slow. You get the full performance of the optimized CUDA and C++ Code. The main problem lies within your graph/network definition:
You can achieve the same goal in different ways, each of them having it’s own (more or less efficient) graph definition. Theoretically it is possible to gain some speed by fusing kernels, optimizing the graph etc. But I must admit, that I don’t know whether there are any of these optimizations already implemented.
In general, running python code gives you a slight overhead (even if the backend is CUDA), because the python interpreter has some stuff like the GIL and some other book-keeping stuff, which is done at compile/linking time for pure C++ programs.