It’s likely that the convs you’re launching are big enough to occupy the entire GPU. When you launch a number of big kernels on different streams, they end up being executed sequentially. Smaller kernels, that use only a small slice of GPU resources, can be parallelized by using multiple streams.