Actually it could be the implementation. If you’re looping over the batch, the CPU is not working in parallel. The batch should always be a dimension in the tensors that are multiplied.
With GPU the implementation can be different because you can make multiple calls and get parallel work. Greater batches are faster on GPU depending on the specs.