title: Dimensions of LSTM variant in Deep Mind's Differentiable Neural Computer (DNC)
body: <p>I'm trying to implement Deep Mind's DNC - <a href="https://www.nature.com/articles/nature20101" rel="nofollow noreferrer">Nature paper</a>- with PyTorch 0.4.0.</p>
  
  <p>When implementing the variant of LSTM they used I encountered some troubles with dimensions.<br>
  To simplify suppose BATCH=1.</p>
  
  <p>The equations they list in the paper are these:
  <a href="https://i.stack.imgur.com/7Q5GH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7Q5GH.png" alt="LSTM variant used by DNC"></a></p>
  
  <p>where <code>[x;h]</code> means a concatenation of <code>x</code> and <code>h</code> into one single vector, and <code>i</code>, <code>f</code> and <code>o</code> are column vectors.</p>
  
  <p>My question is about how the state <code>s_t</code> is computed. <br>
  The second addendum is obtained by multiplying <code>i</code> with a column vector and so the result is either a scalar (transpose <code>i</code> first, then do scalar product) or wrong (two column vectors multiplied).<br>
  <strong>So the state results in a single scalar...</strong></p>
  
  <p>With the same reasoning the hidden state <code>h_t</code> is a scalar too, but it has to be a column vector.</p>
  
  <p>Obviously I'm wrong somewhere, but I can't figure out where. </p>