Answers.Body: <p>It's purely a naming thing (see: <a href="https://arxiv.org/abs/2109.14320" rel="nofollow noreferrer">the footnote on page 3, for example</a>). Recall that on each time-step, the concatenated input + hidden state from the prev. time-step are incorporated into the cell-state.</p>
  <p><a href="https://i.sstatic.net/7na4qdeK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7na4qdeK.png" alt="From https://en.wikipedia.org/wiki/Long_short-term_memory#/media/File:LSTM_Cell.svg" /></a>
  (Image from: <a href="https://en.wikipedia.org/wiki/Long_short-term_memory#/media/File:LSTM_Cell.svg" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Long_short-term_memory#/media/File:LSTM_Cell.svg</a>)</p>
  <p>This concatenated vector passes through two linear layers in parallel: one with a sigmoid activation and another with a tanh activation. The element-wise product of these two output is then added to the cell state. The linear layer with the sigmoid activation is often called the &quot;input gate&quot; while the linear layer with the tanh activation is often called the &quot;input modulation gate&quot;, <a href="https://arxiv.org/pdf/2109.14320" rel="nofollow noreferrer">although they're also often collectively called the &quot;input gate&quot;</a>.</p>
  <p>If I were to guess, giving these two layers separate names is for the purpose of consistency: this way, all <code>{input, output, forget} gates</code> are linear layers with sigmoid activations and share the similar purpose of down-weighting unimportant information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-22T16:24:40.883Z
Answers.LastActivityDate: 2024-06-22T16:24:40.883Z
Answers.UserDisplayName: Alexander Wan
Body: <p>Basically, I was learning about LSTMs where I found LSTMs are made up of three gates: The forget gate, input gate and output gate. However, I came across some sources that state there is a fourth gate called hte <code>input modulation gate</code>. Yet, I can't find much information about this gate. Some sources say it's usually omitted when defining the input gates. Other sources say it's part of the input gate as a sort of sub-system.</p>
  <p><strong>What exactly does the input modulation gate do in LSTMs?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-19T13:26:56.593Z
Comments.Text: Welcome to AI SE. It is helpful for other people to cite the sources you are referring to, such that we can use them to understand your question and provide a better answer.
Comments.UserDisplayName: Robin van Hoorn
Tags: <deep-learning><natural-language-processing><recurrent-neural-networks><long-short-term-memory>
Title: What is the exact purpose of input modulation gate in LSTMs?
UserDisplayName: MrIzzat
ThreadText: # What is the exact purpose of input modulation gate in LSTMs?
  
  (Asked by: username_0 on 2024-06-19)
  
  Basically, I was learning about LSTMs where I found LSTMs are made up of three gates: The forget gate, input gate and output gate. However, I came across some sources that state there is a fourth gate called hte `input modulation gate`. Yet, I can't find much information about this gate. Some sources say it's usually omitted when defining the input gates. Other sources say it's part of the input gate as a sort of sub-system.
  
  **What exactly does the input modulation gate do in LSTMs?**
  
  **Tags:** deep-learning, natural-language-processing, recurrent-neural-networks, long-short-term-memory
  
  > **Comments:**
  > **username_1**: Welcome to AI SE. It is helpful for other people to cite the sources you are referring to, such that we can use them to understand your question and provide a better answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-06-22. Score: 0
  
  It's purely a naming thing (see: [the footnote on page 3, for example](https://arxiv.org/abs/2109.14320)). Recall that on each time-step, the concatenated input + hidden state from the prev. time-step are incorporated into the cell-state.
  
  [![From https://en.wikipedia.org/wiki/Long_short-term_memory#/media/File:LSTM_Cell.svg](https://i.sstatic.net/7na4qdeK.png)](https://i.sstatic.net/7na4qdeK.png)
  (Image from: <https://en.wikipedia.org/wiki/Long_short-term_memory#/media/File:LSTM_Cell.svg>)
  
  This concatenated vector passes through two linear layers in parallel: one with a sigmoid activation and another with a tanh activation. The element-wise product of these two output is then added to the cell state. The linear layer with the sigmoid activation is often called the "input gate" while the linear layer with the tanh activation is often called the "input modulation gate", [although they're also often collectively called the "input gate"](https://arxiv.org/pdf/2109.14320).
  
  If I were to guess, giving these two layers separate names is for the purpose of consistency: this way, all `{input, output, forget} gates` are linear layers with sigmoid activations and share the similar purpose of down-weighting unimportant information.
  
  
  ---
  
  