While there is the new (in torch 1.10) cuda graphs API to deal with the very case, I would not worry about it too much until you find your GPU is underutilized from you using Python for loops.
The Python overhead is more or less constant per operation. The Thomas rule of thumb for this is that if your operands have hundreds of elements, the Python overhead is likely not a crucial bottleneck. On the other hand, if you have many operands that are much smaller, the mere Tensor administration overhead (in C++, too) is likely to be a problem, too.
Best regards
Thomas