Here’s an interesting plot that illustrates how poorly the CUDA streams are performing within my application. My model is designed such that if the CUDA streams are fully parallelized, the time complexity should drop from cubic to quadratic. However, the orange line (the CUDA streams implementation) still has the same time complexity as the serial version. On the other hand, a single-kernel version that I implemented (green line) has a time complexity much closer to 2.

Sadly, the memory usage goes up quite a bit as a result: