Attention Mechanism
[Encoder Model](https://texonom.com/encoder-model-321d79943c8940fcaac0c9ccca0b5f6f) [Decoder Model](https://texonom.com/decoder-model-36e78a40265c473d90197089aebfa83b) [RNN](https://texonom.com/rnn-f7aad56acb5542b2ac26c2908be4ce16) 


**한 번에 전체 문장을 읽고 문장을 기반으로 각 단어의 표현을 병렬로 계산가능**
 Imagine yourself in a library. You have a specific question (**query**). Books on the shelves have titles on their spines (**keys**) that suggest their content. You compare your question to these titles to decide how relevant each book is, and how much **attention** to give each book. Then, you get the information (**value**) from the relevant books to answer your question.
the query vector points to the current input word (aka context).
**The *****keys***** represent the words in the input sentence. The key vectors help the model understand how each word relates to the context word.**
**Attention is how much weight the query word should give each word in the sentence. This is computed via a dot product between the query vector and all the key vectors**. These dot products then go through a softmax which makes the attention scores (across all keys) sum to 1.
**Each word is also represented by a *****value***** which contains the information of that word. As a result, each context word is now represented by an attention-based weightage of all the words in the sentence.**
### NLP Attention Notion
|Title|
|:-:|
|[Self-Attention](https://texonom.com/self-attention-d06bde44563f455b951d17955d820f77)|
|[Multi-head Attention](https://texonom.com/multi-head-attention-d9dfc39b27494123ae4c81f3b98e50b5)|
|[Cross-Attention](https://texonom.com/cross-attention-6365d422d2bf4be78dac038df3a19aae)|
![](https://www.notion.so/image/https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fsecure.notion-static.com%2F36626d64-ac93-430d-aa3b-11ead7cb27e3%2FUntitled.png?table=block&id=6ce231d7-c03f-4556-b56f-0d2d3032f7a3&cache=v2)

### NLP Attention Usages
|Title|
|:-:|
|[Flash Attention](https://texonom.com/flash-attention-ca8093deb6c648059aff1580b0ddbf68)|
|[Dilated Attention](https://texonom.com/dilated-attention-d9000e3460f547bca3f15ac9b8d7e36c)|
|[Multi Query Attention](https://texonom.com/multi-query-attention-5641aba38a8b47caa3a9f364c13789f1)|
|[Group Query Attentiion](https://texonom.com/group-query-attentiion-d72d2da59fba4e9d937a4ec856dac90f)|
|[PagedAttention](https://texonom.com/pagedattention-abf197357343437681fb878ae5700926)|




> [Some Intuition on Attention and the Transformer](https://eugeneyan.com/writing/attention/)

> [[딥러닝] 언어모델, RNN, GRU, LSTM, Attention, Transformer, GPT, BERT 개념 정리](https://velog.io/@rsj9987/딥러닝-용어정리)

> [16-01 트랜스포머(Transformer)](https://wikidocs.net/31379)





