Transformers, Artificial Intelligence, Generative Ai, Gpt, Data Science.
Spotlight The standard attention used by transformers is scaled dot-product attention. You can imagine attention as a spotlight operators can control. For a given position, attention highlights the parts of a sequence that are most relevant and dims the rest. Dot-product attention accomplishes this