Answers.Body: <p>Self attention only acts on one input sequence. This is actually arguable the most common form of attention seen today, as popularized by transformers. In self attention, the keys and values are the same, thus there is only a single input sequence.</p>
  <p>As described in the paper &quot;<a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Attention is All You Need</a>&quot;</p>
  <blockquote>
  <p>Self-attention, sometimes called intra-attention, is an attention mechanism relating different positions of a single sequence in order to
  compute a representation of the sequence.</p>
  </blockquote>
  <p>If you are unfamiliar with self attention, in addition to the paper linked above, <a href="https://machinelearningmastery.com/the-transformer-attention-mechanism/#:%7E:text=Self%2Dattention%2C%20sometimes%20called%20intra,a%20representation%20of%20the%20sequence." rel="nofollow noreferrer">this</a> blog post may be useful. Also see the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html" rel="nofollow noreferrer">Pytorch transformer layer</a> documentation, which is an example of self attention that only takes a single input sequence.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-28T10:27:04.193Z
Answers.Text: Transformers work by generating keys, values and queries from the same input.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T17:22:18.280Z
Answers.LastActivityDate: 2023-02-26T17:22:18.280Z
Answers.UserDisplayName: chessprogrammer
Body: <p>Are there versions of attention that do not require a key-value pair, but just act on one input? Or does this idea simply not make sense?</p>
  
Tags: <machine-learning><transformer><attention>
Title: Are there versions of attention that do not require a key-value pair, but just act on one input?
UserDisplayName: postnubilaphoebus
ThreadText: # Are there versions of attention that do not require a key-value pair, but just act on one input?
  
  (Asked by: username_0 on 2023-02-26)
  
  Are there versions of attention that do not require a key-value pair, but just act on one input? Or does this idea simply not make sense?
  
  **Tags:** machine-learning, transformer, attention
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-26. Score: 2 | Accepted answer
  
  Self attention only acts on one input sequence. This is actually arguable the most common form of attention seen today, as popularized by transformers. In self attention, the keys and values are the same, thus there is only a single input sequence.
  
  As described in the paper "[Attention is All You Need](https://arxiv.org/abs/1706.03762)"
  
  > Self-attention, sometimes called intra-attention, is an attention mechanism relating different positions of a single sequence in order to
  > compute a representation of the sequence.
  
  If you are unfamiliar with self attention, in addition to the paper linked above, [this](https://machinelearningmastery.com/the-transformer-attention-mechanism/#:%7E:text=Self%2Dattention%2C%20sometimes%20called%20intra,a%20representation%20of%20the%20sequence.) blog post may be useful. Also see the [Pytorch transformer layer](https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html) documentation, which is an example of self attention that only takes a single input sequence.
  
  > **Comments:**
  > **username_2**: Transformers work by generating keys, values and queries from the same input.  
  
  ---
  
  