Answers.Body: <p>As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that :
  <span class="math-container">\begin{equation}
      T = P
  \end{equation}</span></p>
  
  <p><span class="math-container">\begin{equation}
      W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI
  \end{equation}</span></p>
  
  <p>Applying a prototype vector as input :</p>
  
  <p><span class="math-container">\begin{equation}
      \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k
  \end{equation}</span></p>
  
  <p>Because they are orthogonal we have :
  <span class="math-container">\begin{equation}
      \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I})
  \end{equation}</span></p>
  
  <p>where <span class="math-container">$R-Q\cdot I$</span> is the length of vectors</p>
  
  <p>So since ,</p>
  
  <p><span class="math-container">\begin{equation}
      W \cdot p_k= (R - Q \cdot I) \cdot p _k
  \end{equation}</span>
  prototype vectors
  continue to be eigenvectors of the new weight matrix .</p>
  
  <p>It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's
  ability to generalize or may increase the biological
  plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T16:34:44.157Z
Answers.LastActivityDate: 2020-04-16T16:34:44.157Z
Answers.UserDisplayName: estamos
Body: <p>How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from <span class="math-container">$W = PP^T- QI$</span>, where <span class="math-container">$Q$</span> is the number of prototype vectors. </p>
  
  <p>I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.</p>
  
  <p>This is a question from Neural Network Design (2nd Edition) book by
  Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .</p>
  
  <p>Resource : <a href="https://hagan.okstate.edu/NNDesign.pdf" rel="noreferrer">E7.5 p 224-225</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-14T18:23:37.187Z
Comments.Text: I found [this](https://page.mi.fu-berlin.de/rojas/neural/chapter/K12.pdf) but I don't think it's what is needed for my question .
Comments.UserDisplayName: estamos
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-14T18:32:04.917Z
Comments.Text: Some more theory about autoassociative nets can be found [here p 48](http://ce.miau.ac.ir/azmoninfo/Chap_3.pdf) .
Comments.UserDisplayName: estamos
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-15T10:17:32.570Z
Comments.Text: In addition some relative [paper p 16 - 20](http://cis.poly.edu/~mleung/CS6673/s09/PatternAssociation.pdf)
Comments.UserDisplayName: estamos
Tags: <neural-networks><machine-learning><learning-algorithms><hebbian-learning>
Title: How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix?
UserDisplayName: estamos
ThreadText: # How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix?
  
  (Asked by: username_0 on 2019-12-14)
  
  How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.
  
  I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.
  
  This is a question from Neural Network Design (2nd Edition) book by
  Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .
  
  Resource : [E7.5 p 224-225](https://hagan.okstate.edu/NNDesign.pdf)
  
  **Tags:** neural-networks, machine-learning, learning-algorithms, hebbian-learning
  
  > **Comments:**
  > **username_0**: I found [this](https://page.mi.fu-berlin.de/rojas/neural/chapter/K12.pdf) but I don't think it's what is needed for my question .  
  > **username_0**: Some more theory about autoassociative nets can be found [here p 48](http://ce.miau.ac.ir/azmoninfo/Chap_3.pdf) .  
  > **username_0**: In addition some relative [paper p 16 - 20](http://cis.poly.edu/~mleung/CS6673/s09/PatternAssociation.pdf)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-04-16. Score: 0 | Accepted answer
  
  As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that :
  \begin{equation}
  T = P
  \end{equation}
  
  \begin{equation}
  W = PP^T - QI = TP^T - QI = \sum\_{q=1}^Q p\_q p\_q^T - QI
  \end{equation}
  
  Applying a prototype vector as input :
  
  \begin{equation}
  \alpha = W \cdot p\_k = \sum\_{q=1}^Q p\_k p\_q p\_q^T - QIp\_k
  \end{equation}
  
  Because they are orthogonal we have :
  \begin{equation}
  \alpha = p\_k(p\_k^T\cdot p\_k) - Q \cdot I \cdot p\_k = p\_k (p\_k^T\cdot p\_k - Q \cdot I ) = p\_k (\textbf{R - Q $\cdot$ I})
  \end{equation}
  
  where $R-Q\cdot I$ is the length of vectors
  
  So since ,
  
  \begin{equation}
  W \cdot p\_k= (R - Q \cdot I) \cdot p \_k
  \end{equation}
  prototype vectors
  continue to be eigenvectors of the new weight matrix .
  
  It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's
  ability to generalize or may increase the biological
  plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used
  
  
  ---
  
  