NLP, ChatGPT, Deep Learning, Artificial Intelligence, Data Science.
alignment model. Formally, given encoder hidden states $h_1, \ldots, h_T$ and decoder hidden state $s_t$, the attention mechanism computes a context vector $c_t$ as: $e_{ti} = \text{score}(s_t, h_i)$ $\alpha_{ti} = \frac{\exp(e_{ti})}{\sum_{j=1}^T \exp(e_{tj})}$ $c_t = \sum_{i=1}^T \alpha_{ti} h_i$