Answers.Body: <p>The forward prop equation is:</p>
  
  <p><span class="math-container">$$
  Z = (X-A)W - B = XW - AW - B
  $$</span></p>
  
  <p>So the derivatives for <span class="math-container">$Z$</span> w.r.t <span class="math-container">$W$</span>, <span class="math-container">$A$</span>, <span class="math-container">$B$</span> and <span class="math-container">$X$</span> should be:</p>
  
  <p><span class="math-container">$$
  \frac{\partial Z}{\partial W} = X-A  \\
  \frac{\partial Z}{\partial A} = - W  \\
  \frac{\partial Z}{\partial B} = - 1 \\
  \frac{\partial Z}{\partial X} = W
  $$</span></p>
  
  <p>I don't know why he needs the last one though. The first is, like you said, as expected. The other two are wrong, I don't know why he used them in the implementation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-04T09:32:32.023Z
Answers.Text: This is exactly what I thought - yet, I have run their code and it works (very well). So there must be something going on here...
Answers.UserDisplayName: Laksh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-04T09:34:44.560Z
Answers.Text: DB seems fine except for the last term at the end. Is this some sort of more advanced gradient method?
Answers.UserDisplayName: Laksh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-04T12:16:12.897Z
Answers.Text: I've seen some "gradient estimation" techniques for cases where the gradient isn't computable but it doesn't seem to be the case here. I also think they would have stated it, seems like a mistake to me. Now why does it work if it is wrong? Well, to be honest, the most important parameters to get right are the weights $W$, so technically it could work even without the biases... If I were you I'd change the code with the correct gradients and see if maybe it works better.
Answers.UserDisplayName: Djib2011
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-04T13:52:42.053Z
Answers.Text: I have tried running it without the A bias, and it works very well too. Will try the 'correct' backprop equations. Also, I have emailed the author of the paper with this link so hopefully he should respond.
Answers.UserDisplayName: Laksh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-04T09:30:18.523Z
Answers.LastActivityDate: 2019-08-04T09:30:18.523Z
Answers.UserDisplayName: Djib2011
Body: <p>Recently I encountered a variant on the normal linear neural layer architecture: Instead of <span class="math-container">$Z = XW + B$</span>, we now have <span class="math-container">$Z = (X-A)W + B$</span>. So we have a 'pre-bias' <span class="math-container">$A$</span> that affects the activation of the last layer, before multiplication by weights. I don't understand the backpropagation equations for <span class="math-container">$dA$</span> and <span class="math-container">$dB$</span> (<span class="math-container">$dW$</span> is as expected).</p>
  <p>Here is the original paper in which it appeared (although the paper itself isn't actually that relevant): <a href="https://papers.nips.cc/paper/4830-learning-invariant-representations-of-molecules-for-atomization-energy-prediction.pdf" rel="nofollow noreferrer">https://papers.nips.cc/paper/4830-learning-invariant-representations-of-molecules-for-atomization-energy-prediction.pdf</a></p>
  <p>Here is the link to the full code of the neural network: <a href="http://www.quantum-machine.org/code/nn-qm7.tar.gz" rel="nofollow noreferrer">http://www.quantum-machine.org/code/nn-qm7.tar.gz</a></p>
  <pre><code>class Linear(Module):
  
      def __init__(self,m,n):
  
          self.tr = m**.5 / n**.5
          self.lr = 1 / m**.5
          
          self.W = numpy.random.normal(0,1 / m**.5,[m,n]).astype('float32')
          self.A = numpy.zeros([m]).astype('float32')
          self.B = numpy.zeros([n]).astype('float32')
  
      def forward(self,X):
          self.X = X
          Y = numpy.dot(X-self.A,self.W)+self.B
          return Y
  
      def backward(self,DY):
          self.DW = numpy.dot((self.X-self.A).T,DY)
          self.DA = -(self.X-self.A).sum(axis=0)
          self.DB = DY.sum(axis=0) + numpy.dot(self.DA,self.W)
          DX = self.tr * numpy.dot(DY,self.W.T)
          return DX
  
      def update(self,lr):
          self.W -= lr*self.lr*self.DW
          self.B -= lr*self.lr*self.DB
          self.A -= lr*self.lr*self.DA
  
      def average(self,nn,a):
          self.W = a*nn.W + (1-a)*self.W
          self.B = a*nn.B + (1-a)*self.B
          self.A = a*nn.A + (1-a)*self.A
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-04T13:53:26.457Z
Comments.Text: My bad, I will post the link to the webpage that contains the code.
Comments.UserDisplayName: Laksh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-07T09:49:08.293Z
Comments.Text: The pre-bias A doesn't help with anything, as the layer before already have a bias and it had already done the work. This method is just repeating another bias which don't help. The only thing it does is increase chance of overfitting and increase training time.
Comments.UserDisplayName: Clement
Tags: <machine-learning><backpropagation><multilayer-perceptrons>
Title: Backpropagation equation for a variant on the usual Linear Neuron architecture
UserDisplayName: Laksh
ThreadText: # Backpropagation equation for a variant on the usual Linear Neuron architecture
  
  (Asked by: username_0 on 2019-08-04)
  
  Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the last layer, before multiplication by weights. I don't understand the backpropagation equations for $dA$ and $dB$ ($dW$ is as expected).
  Here is the original paper in which it appeared (although the paper itself isn't actually that relevant): https://papers.nips.cc/paper/4830-learning-invariant-representations-of-molecules-for-atomization-energy-prediction.pdf
  Here is the link to the full code of the neural network: http://www.quantum-machine.org/code/nn-qm7.tar.gz
  class Linear(Module):
  
      def __init__(self,m,n):
  
          self.tr = m**.5 / n**.5
          self.lr = 1 / m**.5
          
          self.W = numpy.random.normal(0,1 / m**.5,[m,n]).astype('float32')
          self.A = numpy.zeros([m]).astype('float32')
          self.B = numpy.zeros([n]).astype('float32')
  
      def forward(self,X):
          self.X = X
          Y = numpy.dot(X-self.A,self.W)+self.B
          return Y
  
      def backward(self,DY):
          self.DW = numpy.dot((self.X-self.A).T,DY)
          self.DA = -(self.X-self.A).sum(axis=0)
          self.DB = DY.sum(axis=0) + numpy.dot(self.DA,self.W)
          DX = self.tr * numpy.dot(DY,self.W.T)
          return DX
  
      def update(self,lr):
          self.W -= lr*self.lr*self.DW
          self.B -= lr*self.lr*self.DB
          self.A -= lr*self.lr*self.DA
  
      def average(self,nn,a):
          self.W = a*nn.W + (1-a)*self.W
          self.B = a*nn.B + (1-a)*self.B
          self.A = a*nn.A + (1-a)*self.A
  
  **Tags:** machine-learning, backpropagation, multilayer-perceptrons
  > **Comments:**
  > **username_0**: My bad, I will post the link to the webpage that contains the code.  
  > **username_1**: The pre-bias A doesn't help with anything, as the layer before already have a bias and it had already done the work. This method is just repeating another bias which don't help. The only thing it does is increase chance of overfitting and increase training time.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-08-04. Score: 1
  
  The forward prop equation is:
  
  $$
  Z = (X-A)W - B = XW - AW - B
  $$
  
  So the derivatives for $Z$ w.r.t $W$, $A$, $B$ and $X$ should be:
  
  $$
  \frac{\partial Z}{\partial W} = X-A \\
  \frac{\partial Z}{\partial A} = - W \\
  \frac{\partial Z}{\partial B} = - 1 \\
  \frac{\partial Z}{\partial X} = W
  $$
  
  I don't know why he needs the last one though. The first is, like you said, as expected. The other two are wrong, I don't know why he used them in the implementation.
  
  > **Comments:**
  > **username_0**: This is exactly what I thought - yet, I have run their code and it works (very well). So there must be something going on here...  
  > **username_0**: DB seems fine except for the last term at the end. Is this some sort of more advanced gradient method?  
  > **username_2**: I've seen some "gradient estimation" techniques for cases where the gradient isn't computable but it doesn't seem to be the case here. I also think they would have stated it, seems like a mistake to me. Now why does it work if it is wrong? Well, to be honest, the most important parameters to get right are the weights $W$, so technically it could work even without the biases... If I were you I'd change the code with the correct gradients and see if maybe it works better.  
  > **username_0**: I have tried running it without the A bias, and it works very well too. Will try the 'correct' backprop equations. Also, I have emailed the author of the paper with this link so hopefully he should respond.  
  
  ---
  
  