Answers.Body: <p>I am answering this question based on <em>classical</em> backpropagation through time (BPTT) only.</p>
  <p><em>What should I pass as <span class="math-container">$h_0$</span>? is it a zero vector?</em></p>
  <p><strong>Yes</strong>. We know that <span class="math-container">$h_{t-1}$</span> is the footprint of the first <span class="math-container">$t-1$</span> tokens of the input sequence. For the first time, we need to pass <span class="math-container">$h_0$</span> along with the token <span class="math-container">$x_1$</span>. Since, footprint is not formed yet, we need to pass zero vector only. Check <code>Figure 9.4</code> of your pdf.</p>
  <hr />
  <p><em>Does RNN updates its weight matrices <span class="math-container">$U,W,V$</span> after each token of input <span class="math-container">$x_i$</span> ?</em></p>
  <p><strong>No</strong>, RNN does not update its weight matrices <span class="math-container">$U,W,V$</span> after each token of input <span class="math-container">$x_i$</span>. RNN forward pass generates output sequence <span class="math-container">$y = \{y_i\}_{1}^{n}$</span>  for given input <span class="math-container">$x$</span> and the various weight matrices <span class="math-container">$U, V, W$</span> are shared across all timestamps of the forward pass.</p>
  <hr />
  <p><em>Or updates after passing all tokens of a particular input x?</em></p>
  <p><strong>Yes</strong>, all the weight matrices get updated only after generating the complete <span class="math-container">$y$</span>.</p>
  <p>But since the classical BPTT may take much time and the gradients may vanish gradually, truncated backpropagation through time is generally used. And the updation of weights takes place after generating the partial output only.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-04T03:12:21.880Z
Answers.LastActivityDate: 2021-06-04T03:30:19.513Z
Answers.LastEditDate: 2021-06-04T03:30:19.513Z
Answers.UserDisplayName: hanugm
Body: <p>Consider an input to RNN <span class="math-container">$ x = \{x_i\}_{1}^{n}$</span>. Assume that the length of each input <span class="math-container">$x_i$</span> is k.</p>
  <p>Now, consider the following diagram from p5  of <a href="https://web.stanford.edu/%7Ejurafsky/slp3/9.pdf" rel="nofollow noreferrer">this pdf</a></p>
  <p><a href="https://i.sstatic.net/nhirM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nhirM.png" alt="enter image description here" /></a></p>
  <p>My doubts are:</p>
  <ol>
  <li><p>What should I pass as <span class="math-container">$h_0$</span>? is it a zero vector?</p>
  </li>
  <li><p>Does RNN updates its weight matrices <span class="math-container">$U, W, V$</span> after each token of input <span class="math-container">$x_i$</span> ? Or updates after passing all tokens of a particular input <span class="math-container">$x$</span>?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-05T12:47:38.347Z
Comments.Text: Here you're asking 2 distinct questions, although of course they are both related to RNNs. Generally, I always suggest that people focus on [**one main/specific question per post**](https://meta.stackexchange.com/a/39224). If you have multiple questions, it's better to ask them in separate posts, and make sure you provide the necessary context to understand them in each of the posts.
Comments.UserDisplayName: nbro
Tags: <training><recurrent-neural-networks>
Title: Initial Input $h_0$ for RNN and updation of weights
UserDisplayName: hanugm
ThreadText: # Initial Input $h_0$ for RNN and updation of weights
  
  (Asked by: username_0 on 2021-06-03)
  
  Consider an input to RNN $ x = \{x\_i\}\_{1}^{n}$. Assume that the length of each input $x\_i$ is k.
  
  Now, consider the following diagram from p5 of [this pdf](https://web.stanford.edu/%7Ejurafsky/slp3/9.pdf)
  
  [![enter image description here](https://i.sstatic.net/nhirM.png)](https://i.sstatic.net/nhirM.png)
  
  My doubts are:
  
  1. What should I pass as $h\_0$? is it a zero vector?
  2. Does RNN updates its weight matrices $U, W, V$ after each token of input $x\_i$ ? Or updates after passing all tokens of a particular input $x$?
  
  **Tags:** training, recurrent-neural-networks
  
  > **Comments:**
  > **username_1**: Here you're asking 2 distinct questions, although of course they are both related to RNNs. Generally, I always suggest that people focus on [**one main/specific question per post**](https://meta.stackexchange.com/a/39224). If you have multiple questions, it's better to ask them in separate posts, and make sure you provide the necessary context to understand them in each of the posts.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-06-04. Score: 1
  
  I am answering this question based on *classical* backpropagation through time (BPTT) only.
  
  *What should I pass as $h\_0$? is it a zero vector?*
  
  **Yes**. We know that $h\_{t-1}$ is the footprint of the first $t-1$ tokens of the input sequence. For the first time, we need to pass $h\_0$ along with the token $x\_1$. Since, footprint is not formed yet, we need to pass zero vector only. Check `Figure 9.4` of your pdf.
  
  ---
  
  *Does RNN updates its weight matrices $U,W,V$ after each token of input $x\_i$ ?*
  
  **No**, RNN does not update its weight matrices $U,W,V$ after each token of input $x\_i$. RNN forward pass generates output sequence $y = \{y\_i\}\_{1}^{n}$ for given input $x$ and the various weight matrices $U, V, W$ are shared across all timestamps of the forward pass.
  
  ---
  
  *Or updates after passing all tokens of a particular input x?*
  
  **Yes**, all the weight matrices get updated only after generating the complete $y$.
  
  But since the classical BPTT may take much time and the gradients may vanish gradually, truncated backpropagation through time is generally used. And the updation of weights takes place after generating the partial output only.
  
  
  ---
  
  