Thanks for the nice explanation.
This is a very common use-case, and I almost always see constants being used as constants in the equations instead of constant tensors.
There is also the caching; not familiar with GPU caching, but assuming it is similar to CPU caching, such constants are probably cached if they are frequently accessed.