Towards Data Science, Neural Networks, Deep Learning, Machine Learning, Naturallanguageprocessing.
retrieval process in a database (Source: image created by Author) The Attention mechanism is a neural architecture that mimics this process of retrieval. The attention mechanism measures the similarity between the query q and each key-value ki. This similarity returns a weight for each key value.