Transformers, Deep Learning, Machine Learning, Editors Pick, Data Science.
every token regardless of its context. If we knew which tokens were relevant to the current token, one way of capturing this context would be to create a weighted average — or, more generally, a linear combination — of these embeddings. Let’s consider a simple example of how this could look for one