Answers.Body: <p>Examining the architecture of the DNC <a href="https://www.quora.com/How-does-the-Deepmind-DNC-Differentiable-Neural-Computer-compare-to-LSTMs-and-RNNs" rel="nofollow noreferrer">indeed shows many similarities to the LSTM</a>. Consider the diagram in the DeepMind article that you linked to:</p>
  
  <p><a href="https://i.sstatic.net/NwTfn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NwTfn.png" alt="DeepMind DNC Architecture"></a></p>
  
  <p>Compare this to the LSTM architecture (credit to ananth on SlideShare):</p>
  
  <p><a href="https://i.sstatic.net/4Y0tc.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4Y0tc.jpg" alt="enter image description here"></a></p>
  
  <p>There are some close analogs here:</p>
  
  <ul>
  <li>Much like the LSTM, the DNC will perform some conversion from <strong>input</strong> to fixed-size state vectors (<strong>h</strong> and <strong>c</strong> in the LSTM)</li>
  <li>Likewise, the DNC will perform some conversion from these fixed-size state vectors to potentially arbitrarily-lengthed <strong>output</strong> (in the LSTM we repeatedly sample from our model until we are satisfied/the model indicates we are done)</li>
  <li>The <strong>forget</strong> and <strong>input</strong> gates of the LSTM represent the <strong>write</strong> operation in the DNC ('forgetting' is essentially just zeroing or partially zeroing memory)</li>
  <li>The <strong>output</strong> gate of the LSTM represents the <strong>read</strong> operation in the DNC</li>
  </ul>
  
  <p>However, the DNC is definitely more than an LSTM. Most obviously, it utilizes a larger state which is discretized (addressable) into chunks; this allows it to make the forget gate of the LSTM more binary. By this I mean that the state is not necessarily eroded by some fraction at every time step, whereas in the LSTM (with the sigmoid activation function) it necessarily is. This might reduce the problem of catastrophic forgetting that you mentioned and thus scale better.</p>
  
  <p>The DNC is also novel in the links that it uses between memory. However, this might be a more marginal improvement on the LSTM than it seems if we re-imagine the LSTM with complete neural networks for each gate instead of just a single layer with an activation function (call this a super-LSTM); in this case, we can actually learn any relationship between two slots in memory with a sufficiently powerful network. While I don't know the specifics of the links that DeepMind is suggesting, they imply in the article that they are learning everything just by backpropagating gradients like a regular neural network. Therefore whatever relationship they are encoding in their links should theoretically be learnable by a neural network, and so a sufficiently powerful 'super-LSTM' should be able to capture it.</p>
  
  <p><strong>With all that being said</strong>, it is often the case in deep learning that two models with the same theoretical capability for expressiveness perform vastly different in practice. For example, consider that a recurrent network can be represented as a huge feed-forward network if we just unroll it. Similarly, the convolutional network isn't better than a vanilla neural network because it has some extra capacity for expressiveness; in fact, it is the constraints imposed on its weights that makes it <em>more</em> effective. Thus comparing the expressiveness of two models isn't necessarily a fair comparison of their performance in practice, nor an accurate projection of how well they will scale.</p>
  
  <p>One question I have about the DNC is what happens when it runs out of memory. When a classical computer runs out of memory and another block of memory is requested, programs start crashing (at best). I'm curious to see how DeepMind plans to address this. I assume it will rely on some intelligent cannibalization of memory currently in use. In some sense computers currently do this when an OS requests that applications free up non-critical memory if memory pressure reaches a certain threshold.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-10T16:42:20.013Z
Answers.LastActivityDate: 2017-07-10T16:42:20.013Z
Answers.UserDisplayName: eric.mitchell
Body: <p>DeepMind just published a <a href="https://www.nature.com/articles/nature20101" rel="nofollow noreferrer">paper</a> about a <a href="https://deepmind.com/blog/differentiable-neural-computers/" rel="nofollow noreferrer"><em>differentiable neural computer</em></a>, which basically <em>combines a neural network with a memory</em>.</p>
  <p>The idea is to teach the neural network to create and recall useful explicit memories for a certain task. This complements the abilities of a neural network well, because NNs only store knowledge implicitly in the weights and the information used to work on a single task is only stored in the activation of the network and degrades quickly the more information you add. (<a href="https://en.wikipedia.org/wiki/Long_short-term_memory" rel="nofollow noreferrer">LSTMs</a> are one try to slow down this degradation of short term memories, but it still happens.)</p>
  <p>Now, instead of keeping the necessary information in the activation, they presumably keep the addresses of memory slots for specific information in the activation, so these should also be subject to degradation.</p>
  <p>Why would this approach scale? Shouldn't a somewhat higher number of task-specific information once again overwhelm the network's capability of keeping the addresses of all the appropriate memory slots in its activation?</p>
  
Tags: <deep-learning><papers><scalability><neural-turing-machine>
Title: How would DeepMind's new differentiable neural computer scale?
UserDisplayName: BlindKungFuMaster
ThreadText: # How would DeepMind's new differentiable neural computer scale?
  
  (Asked by: username_0 on 2016-10-13)
  
  DeepMind just published a [paper](https://www.nature.com/articles/nature20101) about a [*differentiable neural computer*](https://deepmind.com/blog/differentiable-neural-computers/), which basically *combines a neural network with a memory*.
  
  The idea is to teach the neural network to create and recall useful explicit memories for a certain task. This complements the abilities of a neural network well, because NNs only store knowledge implicitly in the weights and the information used to work on a single task is only stored in the activation of the network and degrades quickly the more information you add. ([LSTMs](https://en.wikipedia.org/wiki/Long_short-term_memory) are one try to slow down this degradation of short term memories, but it still happens.)
  
  Now, instead of keeping the necessary information in the activation, they presumably keep the addresses of memory slots for specific information in the activation, so these should also be subject to degradation.
  
  Why would this approach scale? Shouldn't a somewhat higher number of task-specific information once again overwhelm the network's capability of keeping the addresses of all the appropriate memory slots in its activation?
  
  **Tags:** deep-learning, papers, scalability, neural-turing-machine
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-07-10. Score: 4
  
  Examining the architecture of the DNC [indeed shows many similarities to the LSTM](https://www.quora.com/How-does-the-Deepmind-DNC-Differentiable-Neural-Computer-compare-to-LSTMs-and-RNNs). Consider the diagram in the DeepMind article that you linked to:
  
  [![DeepMind DNC Architecture](https://i.sstatic.net/NwTfn.png)](https://i.sstatic.net/NwTfn.png)
  
  Compare this to the LSTM architecture (credit to ananth on SlideShare):
  
  [![enter image description here](https://i.sstatic.net/4Y0tc.jpg)](https://i.sstatic.net/4Y0tc.jpg)
  
  There are some close analogs here:
  
  * Much like the LSTM, the DNC will perform some conversion from **input** to fixed-size state vectors (**h** and **c** in the LSTM)
  * Likewise, the DNC will perform some conversion from these fixed-size state vectors to potentially arbitrarily-lengthed **output** (in the LSTM we repeatedly sample from our model until we are satisfied/the model indicates we are done)
  * The **forget** and **input** gates of the LSTM represent the **write** operation in the DNC ('forgetting' is essentially just zeroing or partially zeroing memory)
  * The **output** gate of the LSTM represents the **read** operation in the DNC
  
  However, the DNC is definitely more than an LSTM. Most obviously, it utilizes a larger state which is discretized (addressable) into chunks; this allows it to make the forget gate of the LSTM more binary. By this I mean that the state is not necessarily eroded by some fraction at every time step, whereas in the LSTM (with the sigmoid activation function) it necessarily is. This might reduce the problem of catastrophic forgetting that you mentioned and thus scale better.
  
  The DNC is also novel in the links that it uses between memory. However, this might be a more marginal improvement on the LSTM than it seems if we re-imagine the LSTM with complete neural networks for each gate instead of just a single layer with an activation function (call this a super-LSTM); in this case, we can actually learn any relationship between two slots in memory with a sufficiently powerful network. While I don't know the specifics of the links that DeepMind is suggesting, they imply in the article that they are learning everything just by backpropagating gradients like a regular neural network. Therefore whatever relationship they are encoding in their links should theoretically be learnable by a neural network, and so a sufficiently powerful 'super-LSTM' should be able to capture it.
  
  **With all that being said**, it is often the case in deep learning that two models with the same theoretical capability for expressiveness perform vastly different in practice. For example, consider that a recurrent network can be represented as a huge feed-forward network if we just unroll it. Similarly, the convolutional network isn't better than a vanilla neural network because it has some extra capacity for expressiveness; in fact, it is the constraints imposed on its weights that makes it *more* effective. Thus comparing the expressiveness of two models isn't necessarily a fair comparison of their performance in practice, nor an accurate projection of how well they will scale.
  
  One question I have about the DNC is what happens when it runs out of memory. When a classical computer runs out of memory and another block of memory is requested, programs start crashing (at best). I'm curious to see how DeepMind plans to address this. I assume it will rely on some intelligent cannibalization of memory currently in use. In some sense computers currently do this when an OS requests that applications free up non-critical memory if memory pressure reaches a certain threshold.
  
  
  ---
  
  