Purpose of multiple heads is not to parallelize computations. Multiple heads work similar to convolutional filters, each building it’s own logic. Data split you mentioned is not batch-wise split, basic attention from Attention is all you need splits Q,K,V into separate chunks feeding each chunk to different head.