This is correct with how MultiHeadAttention is implemented in the original paper. Each head acts upon a reduced dimensionality of the original problem such that the computation cost (and also parameters) are comparable to that of one head.
There’s another way of performing MultiHeadAttention where the parameters (and thus computation) scales with the number of heads. It’ll basically mimic running several of Pytorch’s MultiHeadAttention layers, each with one head, in parallel. You’ll need to implement this or find an implementation.