Answers.Body: <p>One can view the hidden states just like stated in the question as made up from a linear increasing counter and the input.
  The <strong>query will be the current position <em>n</em></strong> and <strong>key the position count</strong> in the hidden state <em>t</em>:</p>
  <pre><code>a(t, n) = I(t == n/2) = w_t
  </code></pre>
  <p>where I is the indicator function.
  The output can be seen as the <code>sum over all w_t * x_t = x_t</code>.</p>
  <p>To be strict the <strong>value=x_t</strong> can be stored in the hidden states which would be</p>
  <pre><code>h_t = (x_t, t)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-12T19:05:30.460Z
Answers.LastActivityDate: 2023-06-12T19:05:30.460Z
Answers.UserDisplayName: Daraan
Body: <p>I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely certain how to design it. The goal of this question is to understand both mechanisms better.</p>
  <hr />
  <p>So for example I have <code>(10,20,30)</code> or <code>(10,20,30,40,50)</code> given as input sequence.
  At input 30 the RNN should output 20 at position 50 -&gt; 30 and so forth.</p>
  <hr />
  <p>My idea for the RNNs hidden state is to just increase it by 1. The hidden state h would just be a scalar.
  e.g. (10,20,30) produces the states (1,2,3)</p>
  <p>But now I am stuck as attention should work with the input and the hidden state. What I would need as output would be scored (0,1,0) * (10,20,30) = 20.
  The scoring function I come up with would be <code>s(h, number, i) = 1 if h/2 == i else 0</code>.
  But there I am using the index as an additional parameter / positional encoding and wondering if I can do it without it.</p>
  <hr />
  <p>What could be other approaches to handcraft an RNN with attention to extracting the half-position element of a sequence?</p>
  
Tags: <recurrent-neural-networks><attention>
Title: Handcraft RNN with attention to extract central element
UserDisplayName: Daraan
ThreadText: # Handcraft RNN with attention to extract central element
  
  (Asked by: username_0 on 2023-06-08)
  
  I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely certain how to design it. The goal of this question is to understand both mechanisms better.
  
  ---
  
  So for example I have `(10,20,30)` or `(10,20,30,40,50)` given as input sequence.
  At input 30 the RNN should output 20 at position 50 -> 30 and so forth.
  
  ---
  
  My idea for the RNNs hidden state is to just increase it by 1. The hidden state h would just be a scalar.
  e.g. (10,20,30) produces the states (1,2,3)
  
  But now I am stuck as attention should work with the input and the hidden state. What I would need as output would be scored (0,1,0) \* (10,20,30) = 20.
  The scoring function I come up with would be `s(h, number, i) = 1 if h/2 == i else 0`.
  But there I am using the index as an additional parameter / positional encoding and wondering if I can do it without it.
  
  ---
  
  What could be other approaches to handcraft an RNN with attention to extracting the half-position element of a sequence?
  
  **Tags:** recurrent-neural-networks, attention
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-06-12. Score: 0
  
  One can view the hidden states just like stated in the question as made up from a linear increasing counter and the input.
  The **query will be the current position *n*** and **key the position count** in the hidden state *t*:
  
  ```
  a(t, n) = I(t == n/2) = w_t
  
  ```
  
  where I is the indicator function.
  The output can be seen as the `sum over all w_t * x_t = x_t`.
  
  To be strict the **value=x\_t** can be stored in the hidden states which would be
  
  ```
  h_t = (x_t, t)
  
  ```
  
  
  ---
  
  