Answers.Body: <p><a href="https://i.sstatic.net/0BGU1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0BGU1.png" alt="enter image description here" /></a></p>
  <p>They are not the same always, the activation (as you called it) here is the hidden state (I think it's <code>a</code> in your slide), now you can have the output is the same as the hidden state (for example that may represent a word embedding) but for most problems you need to do more operations on the hidden state to calculate the output.</p>
  <p>We do that because what you need as output in the current step may not be the whole hidden state (which may contatin info that required in future steps), also you may need to project the hidden state, for example if you want to predict a word you need to project the hidden state into a vector in <span class="math-container">$\mathbb{R}^{|V|}$</span> where <span class="math-container">$V$</span> is your vocabulary, and then apply a softmax over it (like in the equations above).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-01T01:04:42.680Z
Answers.Text: So, if I understand correctly, the weight wya(in the image) or V(in eq 10.10) can be considered analogous to inserting a dense layer after the RNN layer.
Answers.UserDisplayName: desert_ranger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-01T09:13:18.817Z
Answers.Text: @desert_ranger Yes.
Answers.UserDisplayName: Kais Hasan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-30T21:03:48.663Z
Answers.LastActivityDate: 2022-06-30T21:03:48.663Z
Answers.UserDisplayName: Kais Hasan
Body: <p>I am watching the Sequence models course taught by Andrew Ng. I am a bit lost on the SimpleRNN lecture. As per the lecture, at each time step, there's an output y from a hidden layer and an input activation from the preceding layer. However, I do not understand what's the difference between the activation and output <span class="math-container">$y$</span> at any time step (except the first).</p>
  <p>As per my understanding -</p>
  <p><span class="math-container">$$y = activation(w*x+b)$$</span></p>
  <p>Therefore <span class="math-container">$y_t$</span> and <span class="math-container">$activation_t$</span> should be the same. I have included a figure to describe my issue.</p>
  <p><a href="https://i.sstatic.net/LNs2D.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LNs2D.png" alt="enter image description here" /></a></p>
  <p>In the picture, what's the difference between <span class="math-container">$\hat{a}^{\langle t \rangle}$</span> and <span class="math-container">$\hat{y}^{\langle t \rangle}$</span>?</p>
  <p>Here's the link of the lesson - <a href="https://www.coursera.org/learn/nlp-sequence-models/lecture/ftkzt/recurrent-neural-network-model" rel="nofollow noreferrer">https://www.coursera.org/learn/nlp-sequence-models/lecture/ftkzt/recurrent-neural-network-model</a></p>
  
Tags: <deep-learning><recurrent-neural-networks>
Title: How does the hidden activation differ from the output, at any time step for a SimpleRNN?
UserDisplayName: desert_ranger
ThreadText: # How does the hidden activation differ from the output, at any time step for a SimpleRNN?
  
  (Asked by: username_0 on 2022-06-29)
  
  I am watching the Sequence models course taught by Andrew Ng. I am a bit lost on the SimpleRNN lecture. As per the lecture, at each time step, there's an output y from a hidden layer and an input activation from the preceding layer. However, I do not understand what's the difference between the activation and output $y$ at any time step (except the first).
  
  As per my understanding -
  
  $$y = activation(w\*x+b)$$
  
  Therefore $y\_t$ and $activation\_t$ should be the same. I have included a figure to describe my issue.
  
  [![enter image description here](https://i.sstatic.net/LNs2D.png)](https://i.sstatic.net/LNs2D.png)
  
  In the picture, what's the difference between $\hat{a}^{\langle t \rangle}$ and $\hat{y}^{\langle t \rangle}$?
  
  Here's the link of the lesson - <https://www.coursera.org/learn/nlp-sequence-models/lecture/ftkzt/recurrent-neural-network-model>
  
  **Tags:** deep-learning, recurrent-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-30. Score: 1 | Accepted answer
  
  [![enter image description here](https://i.sstatic.net/0BGU1.png)](https://i.sstatic.net/0BGU1.png)
  
  They are not the same always, the activation (as you called it) here is the hidden state (I think it's `a` in your slide), now you can have the output is the same as the hidden state (for example that may represent a word embedding) but for most problems you need to do more operations on the hidden state to calculate the output.
  
  We do that because what you need as output in the current step may not be the whole hidden state (which may contatin info that required in future steps), also you may need to project the hidden state, for example if you want to predict a word you need to project the hidden state into a vector in $\mathbb{R}^{|V|}$ where $V$ is your vocabulary, and then apply a softmax over it (like in the equations above).
  
  > **Comments:**
  > **username_0**: So, if I understand correctly, the weight wya(in the image) or V(in eq 10.10) can be considered analogous to inserting a dense layer after the RNN layer.  
  > **username_1**: @username_0 Yes.  
  
  ---
  
  