Ah, as I suspected/feared. I have been using the documentation that you linked.
I assume that the same goes for allocation of small, temporary tensors in each thread?
The problem is that each thread must perform mm and mv operations on these temporary tensors. Splitting each part of the algorithm out into multiple kernel calls would be very difficult and inefficient as the operations required depend largely on the internal logic of each thread.
Back to the drawing board. Thank you.