The problem is that einsum reduces to batch matmul and so copies your data around.
The real solution is to implement a more general contraction. I had a patch using TensorIterators instead a few years ago, but somehow I decided that it would not work on CPU and abandoned it instead of measuring it on GPU. And then it never was enough fun to dust it off.
One easy way would be to try the Keops third party library.
Best regards
Thomas