Answers.Body: <p>I kid you not when I tell you I just started with decoupled networks this morning. But the issue is terribly worded by most computer scientists. The way a decoupled neural interface works is not by sending the error backwards, but by storing it in a form of learned error inside a separate network. So not only do we have our nodes and weights, but also the Synths. The Synths modify the weight tables using past error. So instead of passing error back like the article suggests, it actually is passing it forward in a kind of "hey don't do this." What's really fun is that to you never have to mix up your thinking to reverse a network for backprop if you're using a decoupled network. The amazing part is that it is actually easier to code this decoupled network, than it is to write a "simple" ANN, even better still is that it functions a lot like an LSTM network without all the programming mumbo-jumbo. You can see for yourself two different forms of the same network from <a href="https://github.com/CPU-Destraux/Decoupled-Neural-Interface.git" rel="nofollow noreferrer">here</a>. </p>
  
  <p>The code from the demo in the repository is from <a href="https://towardsdatascience.com/only-numpy-implementing-and-comparing-combination-of-google-brains-decoupled-neural-interfaces-6712e758c1af" rel="nofollow noreferrer">https://towardsdatascience.com/only-numpy-implementing-and-comparing-combination-of-google-brains-decoupled-neural-interfaces-6712e758c1af</a> it was a fundamental resource for me, and it may be for you. It also helps you with loss, or cost.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-03T04:16:49.617Z
Answers.Text: Sorry about the dirty code by the way.
Answers.UserDisplayName: Eric Schwarz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-03T04:15:39.663Z
Answers.LastActivityDate: 2018-10-03T04:15:39.663Z
Answers.UserDisplayName: Eric Schwarz
Body: <p>I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper <a href="https://arxiv.org/pdf/1608.05343.pdf" rel="nofollow noreferrer">Decoupled Neural Interfaces using Synthetic Gradients</a> (2017) by Max Jaderberg et al. As in the paper, the DNI is able to produce a synthetic error gradient that reflects the error with respect the output:</p>
  <p><span class="math-container">$$
  \frac{\partial L}{\partial h_{i}}
  $$</span></p>
  <p>I can then use this to update the current layer's parameters by multiplying by the parameters to get the loss with respect to the parameters:</p>
  <p><span class="math-container">$$
  \frac{\partial L}{\partial \theta}=\frac{\partial L}{\partial h_{i}} * \frac{\partial h_{i}}{\partial \theta}
  $$</span></p>
  <p>In the paper, the layer's model is then updated based on the next layer sending the true error backwards.</p>
  <p>My question is, given that I am able to calculate the error with respect to the current output, how do I use this to calculate the loss with respect to the previous layer's output?</p>
  
Tags: <neural-networks><backpropagation><papers><loss>
Title: How to perform back-propagation in Decoupled Neural Interfaces?
UserDisplayName: Brian Sharp
ThreadText: # How to perform back-propagation in Decoupled Neural Interfaces?
  
  (Asked by: username_0 on 2016-12-30)
  
  I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper [Decoupled Neural Interfaces using Synthetic Gradients](https://arxiv.org/pdf/1608.05343.pdf) (2017) by Max Jaderberg et al. As in the paper, the DNI is able to produce a synthetic error gradient that reflects the error with respect the output:
  
  $$
  \frac{\partial L}{\partial h\_{i}}
  $$
  
  I can then use this to update the current layer's parameters by multiplying by the parameters to get the loss with respect to the parameters:
  
  $$
  \frac{\partial L}{\partial \theta}=\frac{\partial L}{\partial h\_{i}} \* \frac{\partial h\_{i}}{\partial \theta}
  $$
  
  In the paper, the layer's model is then updated based on the next layer sending the true error backwards.
  
  My question is, given that I am able to calculate the error with respect to the current output, how do I use this to calculate the loss with respect to the previous layer's output?
  
  **Tags:** neural-networks, backpropagation, papers, loss
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-03. Score: 0
  
  I kid you not when I tell you I just started with decoupled networks this morning. But the issue is terribly worded by most computer scientists. The way a decoupled neural interface works is not by sending the error backwards, but by storing it in a form of learned error inside a separate network. So not only do we have our nodes and weights, but also the Synths. The Synths modify the weight tables using past error. So instead of passing error back like the article suggests, it actually is passing it forward in a kind of "hey don't do this." What's really fun is that to you never have to mix up your thinking to reverse a network for backprop if you're using a decoupled network. The amazing part is that it is actually easier to code this decoupled network, than it is to write a "simple" ANN, even better still is that it functions a lot like an LSTM network without all the programming mumbo-jumbo. You can see for yourself two different forms of the same network from [here](https://github.com/CPU-Destraux/Decoupled-Neural-Interface.git).
  
  The code from the demo in the repository is from <https://towardsdatascience.com/only-numpy-implementing-and-comparing-combination-of-google-brains-decoupled-neural-interfaces-6712e758c1af> it was a fundamental resource for me, and it may be for you. It also helps you with loss, or cost.
  
  > **Comments:**
  > **username_1**: Sorry about the dirty code by the way.  
  
  ---
  
  