Well, so Tensors in inference mode (in particular w/o gradients) should be not that much slower than numpy. If you find a factor 10 in a reproducible comparison, it probably is worth filing a bug.
I don’t use TF much and so I don’t know if it creates intermediate tensors or can avoid it. If anything, I would look into fusion or create a custom cuda kernel, but that is me.
Best regards
Thomas