Answers.Body: <p>You could just do this; concatenate your input_vector with zero's vector that has the size of your output. Then in the first pass you concatenate with the output instaid of the zero's vector. After that repeat.. At the end just compare (compute the loss) your entire output from t0 to t1 to your target and backprop.</p>
  <p>You might want to look into recurrent layers, these are layers that have connections back to themselves so that the network can learn what to &quot;remember&quot;.
  These have some problems with longer sequences, so the &quot;newer&quot; versions try to deal with that. (LSTM and GRU)
  You can also use attention mechanisms if you're dealing with sequences. (basically you learn what parts of your input sequence to look at given a certain &quot;query&quot;, in your case maybe the last timestep) (generally used in natural language processing) But it's a bit more exotic and complicated.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-05T21:55:13.400Z
Answers.LastActivityDate: 2020-08-05T21:55:13.400Z
Answers.UserDisplayName: hal9000
Answers.Body: <p>You want to look at recurrent neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-05T21:44:33.040Z
Answers.LastActivityDate: 2020-08-05T21:44:33.040Z
Answers.UserDisplayName: David
Body: <p>I am quite new to neural networks. I am trying to implement in Python a neural network having only one hidden layer with <span class="math-container">$N$</span> neurons and <span class="math-container">$1$</span> output layer.</p>
  <p>The point is that I am analyzing time series and would like to use the output layer as the input of the next unit: by feeding the network with the input at time <span class="math-container">$t-1$</span> I obtain the output <span class="math-container">$O_{t-1}$</span> and, in the next step, I would like to use both the input at time <span class="math-container">$t$</span> and <span class="math-container">$O_{t-1}$</span>, introducing a sort of auto-regression. I read that <em>recurrent neural network</em> are suitable to address this issue.</p>
  <p>Anyway I cannot imagine how to implement a network in Keras that involves <em>multilayer recurrence</em>: all the references I found are linked to using the output of a layer as input of the same layer in the next step. Instead, I would like to include the output of the last layer (the output layer) in the inputs of the first hidden layer.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-05T22:07:32.063Z
Comments.Text: You originally tagged your post with "recurrent neural networks", so I guess you are aware of them and know what they are used for. If yes, I don't understand the purpose of this question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-05T22:20:03.893Z
Comments.Text: Sure, after a couple of days spent searching I came across "recurrent neural networks" and I have read some things about that. I just cannot understand how can such a RNN be implemented in Python. All the information I found are about networks in which the output (e.g. the hidden states) of a layer is used as input of the same layer; instead, I would like to create a recurrent structure that involves more layers.
Comments.UserDisplayName: polemical
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-05T22:20:51.663Z
Comments.Text: Well, you should have specified that in your question. I suggest that you edit your post to clarify that you are already aware of RNNs and what your question really is.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-05T22:22:20.430Z
Comments.Text: Sorry, I think you're right. I'll do it
Comments.UserDisplayName: polemical
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T00:14:56.210Z
Comments.Text: @piotor Here's an example implementation of an LSTM, a popular type of RNN, in python: https://github.com/nicodjimenez/lstm/blob/master/lstm.py
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-06T01:02:31.093Z
Comments.Text: Note that, on this site, programming issues are generally off-topic. See https://ai.stackexchange.com/help/on-topic. Your last edit seems to make yours a programming issue. So, you probably should ask your question on Stack Overflow. Can you clarify if this is a programming issue or just an engineering/conceptual issue?
Comments.UserDisplayName: nbro
Tags: <neural-networks><python><keras><recurrent-neural-networks>
Title: Is there a neural network that accepts both the current input and previous output?
UserDisplayName: polemical
ThreadText: # Is there a neural network that accepts both the current input and previous output?
  
  (Asked by: username_0 on 2020-08-05)
  
  I am quite new to neural networks. I am trying to implement in Python a neural network having only one hidden layer with $N$ neurons and $1$ output layer.
  
  The point is that I am analyzing time series and would like to use the output layer as the input of the next unit: by feeding the network with the input at time $t-1$ I obtain the output $O\_{t-1}$ and, in the next step, I would like to use both the input at time $t$ and $O\_{t-1}$, introducing a sort of auto-regression. I read that *recurrent neural network* are suitable to address this issue.
  
  Anyway I cannot imagine how to implement a network in Keras that involves *multilayer recurrence*: all the references I found are linked to using the output of a layer as input of the same layer in the next step. Instead, I would like to include the output of the last layer (the output layer) in the inputs of the first hidden layer.
  
  **Tags:** neural-networks, python, keras, recurrent-neural-networks
  > **Comments:**
  > **username_1**: You originally tagged your post with "recurrent neural networks", so I guess you are aware of them and know what they are used for. If yes, I don't understand the purpose of this question.  
  > **username_0**: Sure, after a couple of days spent searching I came across "recurrent neural networks" and I have read some things about that. I just cannot understand how can such a RNN be implemented in Python. All the information I found are about networks in which the output (e.g. the hidden states) of a layer is used as input of the same layer; instead, I would like to create a recurrent structure that involves more layers.  
  > **username_1**: Well, you should have specified that in your question. I suggest that you edit your post to clarify that you are already aware of RNNs and what your question really is.  
  > **username_0**: Sorry, I think you're right. I'll do it  
  > **username_2**: @piotor Here's an example implementation of an LSTM, a popular type of RNN, in python: https://github.com/nicodjimenez/lstm/blob/master/lstm.py  
  > **username_1**: Note that, on this site, programming issues are generally off-topic. See https://ai.stackexchange.com/help/on-topic. Your last edit seems to make yours a programming issue. So, you probably should ask your question on Stack Overflow. Can you clarify if this is a programming issue or just an engineering/conceptual issue?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2020-08-05. Score: 0 | Accepted answer
  
  You could just do this; concatenate your input\_vector with zero's vector that has the size of your output. Then in the first pass you concatenate with the output instaid of the zero's vector. After that repeat.. At the end just compare (compute the loss) your entire output from t0 to t1 to your target and backprop.
  
  You might want to look into recurrent layers, these are layers that have connections back to themselves so that the network can learn what to "remember".
  These have some problems with longer sequences, so the "newer" versions try to deal with that. (LSTM and GRU)
  You can also use attention mechanisms if you're dealing with sequences. (basically you learn what parts of your input sequence to look at given a certain "query", in your case maybe the last timestep) (generally used in natural language processing) But it's a bit more exotic and complicated.
  
  
  ---
  
  ### Answer by username_4 on 2020-08-05. Score: 1
  
  You want to look at recurrent neural networks.
  
  
  ---
  
  