Machine Learning, Transformers, Ashish Vaswani, Lstm, Recurrent Neural Network.
generate keys and queries can be thought of as features of the network. It maps the embeddings to a lower dimensional space where we can use the dot product to calculate the similarity between keys and queries. Then again we use a linear transformation using values. The square root factor in the