Thank you @tevaru for linking this.
To also answer the original question: The fallback_function is a function inserted by the PyTorch JIT profiling executor for cases when its optimization assumptions fail. And that probably is the part of the problem with the performance, too…
So as described in the issue disabling the JIT profiling executor / tensor expression fuser one way or the other helps.