What you see is two things

Even before the JIT, and for “real world” models, the speedup of C++ over Python is smaller than people think as the Python overhead is “fixed cost”. For the LLTM tutorial example, I measured this to be ~10%. For convolutional models like ResNet, it should be less.
One would expect Python -> C++ to be more relevant at speeding things up when you have Python in a “hot loop”. But then you also have Tensors in a hot loop, which bring significant overhead to the point where you’re doing something wrong.
The JIT will remove the Python overheads (including GIL) within the execution of the model, also when called from Python. ONNXRuntime works similar.

The other truth is that you can get further speedups by combining (“fusing”) certain operations to avoid writing intermediates to memory. The pointwise part of an LSTM cell is the classic example. For this you need a “holistic” view of the computation to be able to do optimization across operators.  This is something that the JIT has been doing to some extend (mostly on CUDA) and is the goal of ONNXRuntime/TVM and current PyTorch JIT developments. It would be, however, something that is again independent of C++/Python to invoke the calculation.
Best regards
Thomas