<p>One can view the hidden states just like stated in the question as made up from a linear increasing counter and the input.
The <strong>query will be the current position <em>n</em></strong> and <strong>key the position count</strong> in the hidden state <em>t</em>:</p>
<pre><code>a(t, n) = I(t == n/2) = w_t
</code></pre>
<p>where I is the indicator function.
The output can be seen as the <code>sum over all w_t * x_t = x_t</code>.</p>
<p>To be strict the <strong>value=x_t</strong> can be stored in the hidden states which would be</p>
<pre><code>h_t = (x_t, t)
</code></pre>

CC BY-SA 4.0
2023-06-12T19:05:30.460Z
2023-06-12T19:05:30.460Z
Daraan
<p>I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely certain how to design it. The goal of this question is to understand both mechanisms better.</p>
<hr />
<p>So for example I have <code>(10,20,30)</code> or <code>(10,20,30,40,50)</code> given as input sequence.
At input 30 the RNN should output 20 at position 50 -&gt; 30 and so forth.</p>
<hr />
<p>My idea for the RNNs hidden state is to just increase it by 1. The hidden state h would just be a scalar.
e.g. (10,20,30) produces the states (1,2,3)</p>
<p>But now I am stuck as attention should work with the input and the hidden state. What I would need as output would be scored (0,1,0) * (10,20,30) = 20.
The scoring function I come up with would be <code>s(h, number, i) = 1 if h/2 == i else 0</code>.
But there I am using the index as an additional parameter / positional encoding and wondering if I can do it without it.</p>
<hr />
<p>What could be other approaches to handcraft an RNN with attention to extracting the half-position element of a sequence?</p>

<recurrent-neural-networks><attention>
Handcraft RNN with attention to extract central element
Daraan
# Handcraft RNN with attention to extract central element

(Asked by: username_0 on 2023-06-08)

I am trying to formulate an RNN that uses attention to easily detect the central element of a sequence. For an RNN alone this is not an easy task but with attention, it should be but I am not entirely certain how to design it. The goal of this question is to understand both mechanisms better.

---

So for example I have `(10,20,30)` or `(10,20,30,40,50)` given as input sequence.
At input 30 the RNN should output 20 at position 50 -> 30 and so forth.

---

My idea for the RNNs hidden state is to just increase it by 1. The hidden state h would just be a scalar.
e.g. (10,20,30) produces the states (1,2,3)

But now I am stuck as attention should work with the input and the hidden state. What I would need as output would be scored (0,1,0) \* (10,20,30) = 20.
The scoring function I come up with would be `s(h, number, i) = 1 if h/2 == i else 0`.
But there I am using the index as an additional parameter / positional encoding and wondering if I can do it without it.

---

What could be other approaches to handcraft an RNN with attention to extracting the half-position element of a sequence?

**Tags:** recurrent-neural-networks, attention


---

## 1 answers

### Answer by username_0 on 2023-06-12. Score: 0

One can view the hidden states just like stated in the question as made up from a linear increasing counter and the input.
The query will be the current position n and key the position count in the hidden state t:
a(t, n) = I(t == n/2) = w_t

where I is the indicator function.
The output can be seen as the sum over all w_t * x_t = x_t.
To be strict the value=x_t can be stored in the hidden states which would be
h_t = (x_t, t)


---

