<p>Turns out the reason is because, for places where a dot is shown in the image above, they're actually element-wise multiplications, not dot products. A lot of sources use an X or . to denote multiplication, but don't clearly indicate they mean element-wise multiplication.</p>

CC BY-SA 4.0
2020-10-20T12:49:29.503Z
2020-10-20T12:49:29.503Z
THAT_AI_GUY
<p><a href="https://i.sstatic.net/Xvf5w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Xvf5w.png" alt="enter image description here" /></a></p>
<p>In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance:</p>
<p>The hidden state at timestep t-1 (<span class="math-container">$h_{t-1}$</span>) has shape <span class="math-container">$(na, m)$</span></p>
<p>The input data at timestep t (<span class="math-container">$x_{t}$</span>) has shape <span class="math-container">$(nx, m)$</span></p>
<p><span class="math-container">$Z_{t}$</span> has shape <span class="math-container">$(na+nx, m)$</span> since the hidden state and input data are concatenated in LSTMs.</p>
<p><span class="math-container">$W_{c}$</span> has shape <span class="math-container">$(na, na+nx)$</span></p>
<p><span class="math-container">$W_{c}$</span> • <span class="math-container">$Z_{t}$</span> has shape <span class="math-container">$(na, m)$</span> = <span class="math-container">$i_{t}$</span></p>
<p><span class="math-container">$W_{i}$</span> • <span class="math-container">$Z_{t}$</span> has shape <span class="math-container">$(na, m)$</span> = <span class="math-container">$ĉ_{t}$</span></p>
<p>When working through the network to the point <span class="math-container">$i_{t}$</span> and <span class="math-container">$ĉ_{t}$</span>, how can these two be dot producted when the multiplication is not of the form (m x n)(n x p) as per the matrix multiplication definition?:</p>
<p><a href="https://i.sstatic.net/ChwZK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ChwZK.png" alt="enter image description here" /></a></p>

<neural-networks><machine-learning><recurrent-neural-networks><long-short-term-memory>
How do LSTMs work if the following two matrices are not able to be multiplied?
THAT_AI_GUY
# How do LSTMs work if the following two matrices are not able to be multiplied?

(Asked by: username_0 on 2020-10-20)

[![enter image description here](https://i.sstatic.net/Xvf5w.png)](https://i.sstatic.net/Xvf5w.png)

In the above diagram, the shape of some of the matrices can be seen in the yellow highlight. For instance:

The hidden state at timestep t-1 ($h\_{t-1}$) has shape $(na, m)$

The input data at timestep t ($x\_{t}$) has shape $(nx, m)$

$Z\_{t}$ has shape $(na+nx, m)$ since the hidden state and input data are concatenated in LSTMs.

$W\_{c}$ has shape $(na, na+nx)$

$W\_{c}$ • $Z\_{t}$ has shape $(na, m)$ = $i\_{t}$

$W\_{i}$ • $Z\_{t}$ has shape $(na, m)$ = $ĉ\_{t}$

When working through the network to the point $i\_{t}$ and $ĉ\_{t}$, how can these two be dot producted when the multiplication is not of the form (m x n)(n x p) as per the matrix multiplication definition?:

[![enter image description here](https://i.sstatic.net/ChwZK.png)](https://i.sstatic.net/ChwZK.png)

**Tags:** neural-networks, machine-learning, recurrent-neural-networks, long-short-term-memory


---

## 1 answers

### Answer by username_0 on 2020-10-20. Score: 1

Turns out the reason is because, for places where a dot is shown in the image above, they're actually element-wise multiplications, not dot products. A lot of sources use an X or . to denote multiplication, but don't clearly indicate they mean element-wise multiplication.


---

