Machine Learning, Naturallanguageprocessing, Deep Learning, Mathematics.
applies the softmax function on the dot product of query and keys scaled by 1/sqrt(d_k) to obtain the weights for attention. By using the dot product operation, and using 8 attention head in parallel, the Transformer model becomes much more computationally efficient than its recurrent model