Actually Pytorch uses something similar prior to CUDA 8.0, and it’s still effective here 6. But as you said, allocating lots of pointers is not that efficient. For the toy example above, it’s allocating 30K pointers and actually slower than making copies +  gemmStridedBatched. It would be really nice if we can do gemmStridedBatched without copying.