Answers.Body: <p>Firstly, I am assuming <span class="math-container">$r$</span> is the update to the hidden state. I will call it <span class="math-container">$\tilde{\mathbf{h}}_t$</span>, because this is how it's called in the paper by <a href="https://arxiv.org/abs/1406.1078v3" rel="nofollow noreferrer">Cho et al.</a> that introduced the GRU (where <span class="math-container">$\mathbf{r}_t$</span> is actually a different vector).</p>
  <p>Both formulas do exactly the same thing. The only difference is how you would interpret the role of the vector <span class="math-container">$\mathbf{z}_t \in (0, 1)^d$</span>. Importantly, because <span class="math-container">$\mathbf{z}_t$</span> is sigmoid-activated, it's values are between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>. Thus, you could say that <span class="math-container">$\mathbf{z}_t$</span> acts like a continuous switch that controls how much of <span class="math-container">$\mathbf{h}_{t-1}$</span> and <span class="math-container">$\tilde{\mathbf{h}}_t$</span> ends up in the updated hidden state <span class="math-container">$\mathbf{h}_t$</span>. This is also called a gate.</p>
  <p>In the first formula <span class="math-container">$\mathbf{h}_t = \mathbf{h}_{t-1}\mathbf{z}_t+\tilde{\mathbf{h}}_t(1-\mathbf{z}_t)$</span><br>
  you could say that <span class="math-container">$\mathbf{z}_t$</span> acts like a <em>keep</em>-gate that tells you how much of the old information in the hidden state is preserved. It's negative (<span class="math-container">$1  - \mathbf{z}_t$</span>), tells you how much the new information <span class="math-container">$\tilde{\mathbf{h}}_t$</span> contributes.</p>
  <p>In the second formula: <span class="math-container">$\mathbf{h}_t = \mathbf{h}_{t-1}(1 - \mathbf{z}_t)+\tilde{\mathbf{h}}_t\mathbf{z}_t$</span><br>
  the role of <span class="math-container">$\mathbf{z}_t$</span> is inverted and could now be interpreted as an <em>update</em>-gate, controlling how much of old information is replaced by the update <span class="math-container">$\tilde{\mathbf{h}}_t$</span>.</p>
  <p>From an optimization perspective, it doesn't matter whether you optimize the negative or the positive version (i.e. the <em>keep</em> or the <em>update</em> version) of <span class="math-container">$\mathbf{z}_t$</span>. Therefore, both equations are valid and work equally well.
  The first formula is actually used in the original paper by Cho et al., where it is called an update gate (I would argue that z, in this case, acts rather like a <em>keep</em>-gate, but in the end its negative controls the portion of the update, so I guess you can also call it that).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-04T12:45:35.783Z
Answers.Text: Oh okay that makes sense. Thank you for this really good explanation.
Answers.UserDisplayName: Johannes K.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-04T09:52:33.043Z
Answers.LastActivityDate: 2023-01-04T09:52:33.043Z
Answers.UserDisplayName: Chillston
Body: <p>Im doing trying to implement GRU in my own Neural Network Library but when I did some research i stumbled on some inconsistencies.</p>
  <p>When calculating a cell there are as many legitimate resources which state that <span class="math-container">$\mathbf{h}_t = \mathbf{h}_{t-1}\mathbf{z}_t+\mathbf{r}_t(1-\mathbf{z}_t)$</span> (e.g. <a href="https://arxiv.org/pdf/1803.01686.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1803.01686.pdf</a>, <a href="https://d2l.ai/chapter_recurrent-modern/gru.html" rel="nofollow noreferrer">https://d2l.ai/chapter_recurrent-modern/gru.html</a>) as there are which state <span class="math-container">$\mathbf{h}_t = \mathbf{h}_{t-1}(1 - \mathbf{z}_t)+\mathbf{r}_t\mathbf{z}_t$</span>. (e.g. <a href="https://arxiv.org/pdf/2107.02248.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2107.02248.pdf</a>, <a href="http://dprogrammer.org/rnn-lstm-gru" rel="nofollow noreferrer">http://dprogrammer.org/rnn-lstm-gru</a>) When looking at diagrams there are also inconsistencies between the formulas and the diagrams themself. Also the diagrams differ.
  Can someone tell me which one it is and why?</p>
  
Tags: <machine-learning><recurrent-neural-networks><gated-recurrent-unit>
Title: Which calculation to use for GRU
UserDisplayName: Johannes K.
ThreadText: # Which calculation to use for GRU
  
  (Asked by: username_0 on 2023-01-04)
  
  Im doing trying to implement GRU in my own Neural Network Library but when I did some research i stumbled on some inconsistencies.
  
  When calculating a cell there are as many legitimate resources which state that $\mathbf{h}\_t = \mathbf{h}\_{t-1}\mathbf{z}\_t+\mathbf{r}\_t(1-\mathbf{z}\_t)$ (e.g. <https://arxiv.org/pdf/1803.01686.pdf>, <https://d2l.ai/chapter_recurrent-modern/gru.html>) as there are which state $\mathbf{h}\_t = \mathbf{h}\_{t-1}(1 - \mathbf{z}\_t)+\mathbf{r}\_t\mathbf{z}\_t$. (e.g. <https://arxiv.org/pdf/2107.02248.pdf>, <http://dprogrammer.org/rnn-lstm-gru>) When looking at diagrams there are also inconsistencies between the formulas and the diagrams themself. Also the diagrams differ.
  Can someone tell me which one it is and why?
  
  **Tags:** machine-learning, recurrent-neural-networks, gated-recurrent-unit
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-04. Score: 2 | Accepted answer
  
  Firstly, I am assuming $r$ is the update to the hidden state. I will call it $\tilde{\mathbf{h}}\_t$, because this is how it's called in the paper by [Cho et al.](https://arxiv.org/abs/1406.1078v3) that introduced the GRU (where $\mathbf{r}\_t$ is actually a different vector).
  
  Both formulas do exactly the same thing. The only difference is how you would interpret the role of the vector $\mathbf{z}\_t \in (0, 1)^d$. Importantly, because $\mathbf{z}\_t$ is sigmoid-activated, it's values are between $0$ and $1$. Thus, you could say that $\mathbf{z}\_t$ acts like a continuous switch that controls how much of $\mathbf{h}\_{t-1}$ and $\tilde{\mathbf{h}}\_t$ ends up in the updated hidden state $\mathbf{h}\_t$. This is also called a gate.
  
  In the first formula $\mathbf{h}\_t = \mathbf{h}\_{t-1}\mathbf{z}\_t+\tilde{\mathbf{h}}\_t(1-\mathbf{z}\_t)$  
  you could say that $\mathbf{z}\_t$ acts like a *keep*-gate that tells you how much of the old information in the hidden state is preserved. It's negative ($1 - \mathbf{z}\_t$), tells you how much the new information $\tilde{\mathbf{h}}\_t$ contributes.
  
  In the second formula: $\mathbf{h}\_t = \mathbf{h}\_{t-1}(1 - \mathbf{z}\_t)+\tilde{\mathbf{h}}\_t\mathbf{z}\_t$  
  the role of $\mathbf{z}\_t$ is inverted and could now be interpreted as an *update*-gate, controlling how much of old information is replaced by the update $\tilde{\mathbf{h}}\_t$.
  
  From an optimization perspective, it doesn't matter whether you optimize the negative or the positive version (i.e. the *keep* or the *update* version) of $\mathbf{z}\_t$. Therefore, both equations are valid and work equally well.
  The first formula is actually used in the original paper by Cho et al., where it is called an update gate (I would argue that z, in this case, acts rather like a *keep*-gate, but in the end its negative controls the portion of the update, so I guess you can also call it that).
  
  > **Comments:**
  > **username_0**: Oh okay that makes sense. Thank you for this really good explanation.  
  
  ---
  
  