You typically have two forms of parallelism on the CPU, intra-op (so one operation is executed on several CPUs) and inter-op (so several operations are launched in parallel).
You might see that the first works better for larger batches.