The CPU kernel is slow because we use a for loop 5 over batch dimension. I think the main reason for making copies and utilizing  bmm is that we don’t have a readily available kernel for the GPU op otherwise. Right now CUDA provides cublasSgemmBatched, cublasDgemmBatched and their strided versions for batched mm.