After chatting with some cuda experts, the answer is:
For small 20x20 operations, the overhead of the multistream operation will be larger than running everything on the same stream. There’s a narrow  zone where streams actually help, but 20x20 is not it, things like that should be handled by batches kernels.