Answers.Body: <p>Turns out the reason is because, for places where a dot is shown in the image above, they're actually element-wise multiplications, not dot products. A lot of sources use an X or . to denote multiplication, but don't clearly indicate they mean element-wise multiplication.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T12:49:29.503Z
Answers.LastActivityDate: 2020-10-20T12:49:29.503Z
Answers.UserDisplayName: THAT_AI_GUY
Body: <p><a href="https://i.sstatic.net/Xvf5w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Xvf5w.png" alt="enter image description here" /></a></p>
  <p>In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance:</p>
  <p>The hidden state at timestep t-1 (<span class="math-container">$h_{t-1}$</span>) has shape <span class="math-container">$(na, m)$</span></p>
  <p>The input data at timestep t (<span class="math-container">$x_{t}$</span>) has shape <span class="math-container">$(nx, m)$</span></p>
  <p><span class="math-container">$Z_{t}$</span> has shape <span class="math-container">$(na+nx, m)$</span> since the hidden state and input data are concatenated in LSTMs.</p>
  <p><span class="math-container">$W_{c}$</span> has shape <span class="math-container">$(na, na+nx)$</span></p>
  <p><span class="math-container">$W_{c}$</span> • <span class="math-container">$Z_{t}$</span> has shape <span class="math-container">$(na, m)$</span> = <span class="math-container">$i_{t}$</span></p>
  <p><span class="math-container">$W_{i}$</span> • <span class="math-container">$Z_{t}$</span> has shape <span class="math-container">$(na, m)$</span> = <span class="math-container">$ĉ_{t}$</span></p>
  <p>When working through the network to the point <span class="math-container">$i_{t}$</span> and <span class="math-container">$ĉ_{t}$</span>, how can these two be dot producted when the multiplication is not of the form (m x n)(n x p) as per the matrix multiplication definition?:</p>
  <p><a href="https://i.sstatic.net/ChwZK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ChwZK.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><machine-learning><recurrent-neural-networks><long-short-term-memory>
Title: How do LSTMs work if the following two matrices are not able to be multiplied?
UserDisplayName: THAT_AI_GUY
ThreadText: # How do LSTMs work if the following two matrices are not able to be multiplied?
  
  (Asked by: username_0 on 2020-10-20)
  
  [![enter image description here](https://i.sstatic.net/Xvf5w.png)](https://i.sstatic.net/Xvf5w.png)
  
  In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance:
  
  The hidden state at timestep t-1 ($h\_{t-1}$) has shape $(na, m)$
  
  The input data at timestep t ($x\_{t}$) has shape $(nx, m)$
  
  $Z\_{t}$ has shape $(na+nx, m)$ since the hidden state and input data are concatenated in LSTMs.
  
  $W\_{c}$ has shape $(na, na+nx)$
  
  $W\_{c}$ • $Z\_{t}$ has shape $(na, m)$ = $i\_{t}$
  
  $W\_{i}$ • $Z\_{t}$ has shape $(na, m)$ = $ĉ\_{t}$
  
  When working through the network to the point $i\_{t}$ and $ĉ\_{t}$, how can these two be dot producted when the multiplication is not of the form (m x n)(n x p) as per the matrix multiplication definition?:
  
  [![enter image description here](https://i.sstatic.net/ChwZK.png)](https://i.sstatic.net/ChwZK.png)
  
  **Tags:** neural-networks, machine-learning, recurrent-neural-networks, long-short-term-memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-10-20. Score: 1
  
  Turns out the reason is because, for places where a dot is shown in the image above, they're actually element-wise multiplications, not dot products. A lot of sources use an X or . to denote multiplication, but don't clearly indicate they mean element-wise multiplication.
  
  
  ---
  
  