where K {\displaystyle K} is the number of attention heads, ‖ {\displaystyle {\Big \Vert }} denotes vector concatenation, σ ( ⋅ ) {\displaystyle \sigma (\cdot )} is an activation function (e.g., ReLU), α i j {\displaystyle \alpha _{ij}} are attention coefficients, and W k {\displaystyle W^{k}} is a matrix of trainable parameters for the k {\displaystyle k} -th attention head.