Answers.Body: <p>Congrats, you have invented 1d convolution. Convolution combined with RNN would have some advantage over just RNN. Think about the perception field. 
  In this layer, you do aggregate <span class="math-container">$6$</span> values to one. Imagine two of them - it will be <span class="math-container">$36$</span> already, etc. But, in the end, you still need RNN at the end to aggregate a variable length to constant length.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T19:14:41.520Z
Answers.Text: Well that's good! Glad I'm on the right track! (Not sure what you mean at the end about variable lengths).
Answers.UserDisplayName: zooby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T19:40:32.190Z
Answers.Text: @zooby This is not a 1D CNN, its a non differentiable RNN. (actions must be sampled under some categorical distribution based on whats described). The only similarity to a 1d cnn is the sliding window
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T19:42:28.150Z
Answers.Text: Why is it non-differentiable ?
Answers.UserDisplayName: zooby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T20:14:18.503Z
Answers.Text: do you train with sequences of different lenght, right? also if you put output as input think about output may be wrong so you can consider to force-feeding ( expected data instead of output)
Answers.UserDisplayName: user8426627
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T20:30:34.163Z
Answers.Text: I could be wrong but generally actions are drawn from a distribution (that’s why you show one hot encodingns) and you can’t differentiate through a categorical distrib
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-27T18:50:55.280Z
Answers.LastActivityDate: 2019-07-27T22:33:09.400Z
Answers.LastEditDate: 2019-07-27T22:33:09.400Z
Answers.UserDisplayName: user8426627
Body: <p>I want to design a NN that can remember it's last 7 actions and use them as inputs. So for example it would be able to store words in it's memory. Therefore if it had a choice of 10 different actions, the number of words it could store is <span class="math-container">$10^7$</span>.</p>
  
  <p>Here is my design:</p>
  
  <p><span class="math-container">$$out_{n+1} = f(out_n, in_n)\mathbf{N} + out_n.\mathbf{M}$$</span></p>
  
  <p><span class="math-container">$$action_n = \sigma(\mathbf{N} \cdot out_n)$$</span> </p>
  
  <p>Where <span class="math-container">$f$</span> represents some layered neural network. Some of the actions would be physical actions and some might be internal (such as thinking of the letter 'C').</p>
  
  <p>Basically I want <span class="math-container">$out_n$</span> to be an array that keeps the last 6 action values and puts them back in. So <span class="math-container">$M$</span> will be the matrix:</p>
  
  <p><span class="math-container">$$\begin{bmatrix}
  0&amp;1&amp;0&amp;0&amp;0&amp;0\\
  0&amp;0&amp;1&amp;0&amp;0&amp;0\\
  0&amp;0&amp;0&amp;1&amp;0&amp;0\\
  0&amp;0&amp;0&amp;0&amp;1&amp;0\\
  0&amp;0&amp;0&amp;0&amp;0&amp;1\\
  0&amp;0&amp;0&amp;0&amp;0&amp;0
  \end{bmatrix}$$</span></p>
  
  <p>i.e. it would drop the 6th item from it's memory.</p>
  
  <p>and <span class="math-container">$N$</span> would be the vector:</p>
  
  <p><span class="math-container">$$\begin{bmatrix}
  1&amp;0&amp;0&amp;0&amp;0&amp;0&amp;0
  \end{bmatrix}$$</span></p>
  
  <p>I think this would be equivalent to an equation of the form:</p>
  
  <p><span class="math-container">$$out_{n+1}=F(in_n,out_n,out_{n-1},out_{n-2},...,out_{n-6})$$</span></p>
  
  <p>So I think this would be an advantage over an RNN since this model remembers precisely it's last 6 actions. But would this be better than an RNN or worse? One could increase it's memory to more than 7 quite easily. </p>
  
  <p>I think it's basically the same archececture as an RNN except elinimating a lot of the connections. Is this a new design or a common design?</p>
  
  <p>One problem with this design is that you might also want a memory that is over longer time periods (e.g. for actions that take more than one tick.) But that might be solved by enhancing the archecture.</p>
  
Tags: <neural-networks><long-short-term-memory>
Title: Would this neural network have short term memory?
UserDisplayName: zooby
ThreadText: # Would this neural network have short term memory?
  
  (Asked by: username_0 on 2019-07-27)
  
  I want to design a NN that can remember it's last 7 actions and use them as inputs. So for example it would be able to store words in it's memory. Therefore if it had a choice of 10 different actions, the number of words it could store is $10^7$.
  
  Here is my design:
  
  $$out\_{n+1} = f(out\_n, in\_n)\mathbf{N} + out\_n.\mathbf{M}$$
  
  $$action\_n = \sigma(\mathbf{N} \cdot out\_n)$$
  
  Where $f$ represents some layered neural network. Some of the actions would be physical actions and some might be internal (such as thinking of the letter 'C').
  
  Basically I want $out\_n$ to be an array that keeps the last 6 action values and puts them back in. So $M$ will be the matrix:
  
  $$\begin{bmatrix}
  0&1&0&0&0&0\\
  0&0&1&0&0&0\\
  0&0&0&1&0&0\\
  0&0&0&0&1&0\\
  0&0&0&0&0&1\\
  0&0&0&0&0&0
  \end{bmatrix}$$
  
  i.e. it would drop the 6th item from it's memory.
  
  and $N$ would be the vector:
  
  $$\begin{bmatrix}
  1&0&0&0&0&0&0
  \end{bmatrix}$$
  
  I think this would be equivalent to an equation of the form:
  
  $$out\_{n+1}=F(in\_n,out\_n,out\_{n-1},out\_{n-2},...,out\_{n-6})$$
  
  So I think this would be an advantage over an RNN since this model remembers precisely it's last 6 actions. But would this be better than an RNN or worse? One could increase it's memory to more than 7 quite easily.
  
  I think it's basically the same archececture as an RNN except elinimating a lot of the connections. Is this a new design or a common design?
  
  One problem with this design is that you might also want a memory that is over longer time periods (e.g. for actions that take more than one tick.) But that might be solved by enhancing the archecture.
  
  **Tags:** neural-networks, long-short-term-memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-27. Score: 2
  
  Congrats, you have invented 1d convolution. Convolution combined with RNN would have some advantage over just RNN. Think about the perception field.
  In this layer, you do aggregate $6$ values to one. Imagine two of them - it will be $36$ already, etc. But, in the end, you still need RNN at the end to aggregate a variable length to constant length.
  
  > **Comments:**
  > **username_0**: Well that's good! Glad I'm on the right track! (Not sure what you mean at the end about variable lengths).  
  > **username_2**: @username_0 This is not a 1D CNN, its a non differentiable RNN. (actions must be sampled under some categorical distribution based on whats described). The only similarity to a 1d cnn is the sliding window  
  > **username_0**: Why is it non-differentiable ?  
  > **username_1**: do you train with sequences of different lenght, right? also if you put output as input think about output may be wrong so you can consider to force-feeding ( expected data instead of output)  
  > **username_2**: I could be wrong but generally actions are drawn from a distribution (that’s why you show one hot encodingns) and you can’t differentiate through a categorical distrib  
  
  ---
  
  