If the overhead of launching a stream is so high, would it help if the CUDA streams were launched from different threads?

I have to admit I don’t know. cc @ngimel
Is there any chance you could write your function to be applied on a 1D Tensor that contains the upper triangular part of the matrix? that would reduce the memory and you can use a single kernel.