Body: <p>In <a href="http://page.mi.fu-berlin.de/rojas/neural/chapter/K9.pdf" rel="nofollow noreferrer">Chapter 9</a>, section 9.1.6, Raul Rojas describes how committees of networks can reduce the prediction error by training N identical neural networks and averaging the results.</p>
  <p>If <span class="math-container">$f_i$</span> are the functions approximated by the <span class="math-container">$N$</span> neural nets, then:</p>
  <p><span class="math-container">$$
  Q=\left|\frac{1}{N}(1,1, \ldots, 1) \mathbf{E}\right|^{2}=\frac{1}{N^{2}}(1,1, \ldots, 1) \mathbf{E} \mathbf{E}^{\mathrm{T}}(1,1, \ldots, 1)^{\mathrm{T}}\tag{9.4}\label{9.4}
  $$</span>
  is the quadratic error of the average of the networks, where
  <span class="math-container">$$
  \mathbf{E}=\left(\begin{array}{cccc}
  e_{1}^{1} &amp; e_{2}^{1} &amp; \cdots &amp; e_{m}^{1} \\
  \vdots &amp; \vdots &amp; \ddots &amp; \vdots \\
  e_{1}^{N} &amp; e_{2}^{N} &amp; \cdots &amp; e_{m}^{N}
  \end{array}\right),
  $$</span>
  and <span class="math-container">$\mathbf{E}$</span>'s rows are the errors of the approximations of the <span class="math-container">$N$</span> functions, i.e. <span class="math-container">$\mathbf{e}^{i} = f_i(\mathbf{x}^{i}) - t_i$</span>, for each of the input-output pairs <span class="math-container">$\left(\mathbf{x}^{1}, t_{1}\right), \ldots,\left(\mathbf{x}^{m}, t_{m}\right)$</span> used in training.</p>
  <p>Is there a way to assure that the errors for a neural network are uncorrelated to the errors of the others?</p>
  <p>Raul Rojas says that the uncorrelation of residual errors is true for a not too large <span class="math-container">$N$</span> (i.e. <span class="math-container">$N &lt; 4$</span>). Why is that?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T13:51:18.643Z
Comments.Text: Maybe these notes [https://people.math.umass.edu/~johnpb/s706/notes_mixed.pdf](https://people.math.umass.edu/~johnpb/s706/notes_mixed.pdf) could be useful.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T16:12:20.243Z
Comments.Text: I think that the tag that you actually want to use is [tag:ensemble-learning] and not [tag:committees-of-networks], which I've never heard of, but it's possible it exists or has been used to refer to this approach.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T16:31:01.203Z
Comments.Text: @nbro section 9.1.6 is dedicated to it. Ensemble seems to be mixing different networks, or models. Committees is for identical networks, trained with the same data.
Comments.UserDisplayName: EmmanuelMess
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T16:35:19.870Z
Comments.Text: @nbro thanks for the pdf, but I don't understand how I should use that information to assure uncorrelated errors, is there an algorithm name I can look up (that use the Unconditional or marginal model or the Nonlinear mixed model as those are the nonlinear methods)?
Comments.UserDisplayName: EmmanuelMess
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T16:36:01.780Z
Comments.Text: I don't think that [ensemble learning](http://www.scholarpedia.org/article/Ensemble_learning) is restricted to different networks. Nowadays, it's typically used to indicate that you train multiple models and then combine them somehow, but I'm also not an expert in ensemble learning, to be honest. In any case, I've just looked at that chapter 9.1.6 again and he cites [399], which is a paper entitled ["When Networks Disagree: Ensemble Methods for Hybrid Neural Networks"](https://apps.dtic.mil/sti/pdfs/ADA260045.pdf).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T16:36:04.993Z
Comments.Text: The book you're reading is not very new (although still a good book, as far as I'm concerned), so, at the time of writing that book, maybe "ensemble learning" was not very used yet as a standard term.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T16:39:24.973Z
Comments.Text: Regarding the link above, I don't really know if it could help you. It talks about the correlation of errors in linear regression, so I thought it could be useful, at least, to understand what R. Rojas meant by "correlation". You could also interpret neural networks as performing non-linear regression, so that's another reason why I provided the link to that pdf, which, to be honest, I have only skimmed through.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T20:04:05.050Z
Comments.Text: its a theoretical question about a practical approach. the payoff from this technique would tend to depend in part on the data, the network architecture, the features extracted, etc... in other words, the answer would mostly come from experiments in particular situations. basically there is some randomness in training also in which different nets may "find" or "emphasize" different actual features of the data.
Comments.UserDisplayName: vzn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T20:48:02.163Z
Comments.Text: @vzn Should I try to make the starting weights be far apart from network to network? something like randomly nudge weights so that they are different from the other networks' weights?
Comments.UserDisplayName: EmmanuelMess
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-20T21:38:42.773Z
Comments.Text: yes try starting with different weights and possibly train the nets on different samples of the same data, ie try data segregation ideas. also Rojas was saying that it can work for as low as N=2 or N=3 but that doesnt mean to keep N in that range. the point is N varies for each situation and just test to find what works well/ is practical etc. you can ping me in [chat] or this chat room for further discussion https://chat.stackexchange.com/rooms/9446/theory-salon
Comments.UserDisplayName: vzn
Tags: <neural-networks><proofs><statistics><ensemble-learning>
Title: When do two identical neural networks have uncorrelated errors?
UserDisplayName: EmmanuelMess
ThreadText: # When do two identical neural networks have uncorrelated errors?
  
  (Asked by: username_0 on 2021-01-19)
  
  In [Chapter 9](http://page.mi.fu-berlin.de/rojas/neural/chapter/K9.pdf), section 9.1.6, Raul Rojas describes how committees of networks can reduce the prediction error by training N identical neural networks and averaging the results.
  
  If $f\_i$ are the functions approximated by the $N$ neural nets, then:
  
  $$
  Q=\left|\frac{1}{N}(1,1, \ldots, 1) \mathbf{E}\right|^{2}=\frac{1}{N^{2}}(1,1, \ldots, 1) \mathbf{E} \mathbf{E}^{\mathrm{T}}(1,1, \ldots, 1)^{\mathrm{T}}\tag{9.4}\label{9.4}
  $$
  is the quadratic error of the average of the networks, where
  $$
  \mathbf{E}=\left(\begin{array}{cccc}
  e\_{1}^{1} & e\_{2}^{1} & \cdots & e\_{m}^{1} \\
  \vdots & \vdots & \ddots & \vdots \\
  e\_{1}^{N} & e\_{2}^{N} & \cdots & e\_{m}^{N}
  \end{array}\right),
  $$
  and $\mathbf{E}$'s rows are the errors of the approximations of the $N$ functions, i.e. $\mathbf{e}^{i} = f\_i(\mathbf{x}^{i}) - t\_i$, for each of the input-output pairs $\left(\mathbf{x}^{1}, t\_{1}\right), \ldots,\left(\mathbf{x}^{m}, t\_{m}\right)$ used in training.
  
  Is there a way to assure that the errors for a neural network are uncorrelated to the errors of the others?
  
  Raul Rojas says that the uncorrelation of residual errors is true for a not too large $N$ (i.e. $N < 4$). Why is that?
  
  **Tags:** neural-networks, proofs, statistics, ensemble-learning
  
  > **Comments:**
  > **username_1**: Maybe these notes [https://people.math.umass.edu/~johnpb/s706/notes_mixed.pdf](https://people.math.umass.edu/~johnpb/s706/notes_mixed.pdf) could be useful.  
  > **username_1**: I think that the tag that you actually want to use is [tag:ensemble-learning] and not [tag:committees-of-networks], which I've never heard of, but it's possible it exists or has been used to refer to this approach.  
  > **username_0**: @username_1 section 9.1.6 is dedicated to it. Ensemble seems to be mixing different networks, or models. Committees is for identical networks, trained with the same data.  
  > **username_0**: @username_1 thanks for the pdf, but I don't understand how I should use that information to assure uncorrelated errors, is there an algorithm name I can look up (that use the Unconditional or marginal model or the Nonlinear mixed model as those are the nonlinear methods)?  
  > **username_1**: I don't think that [ensemble learning](http://www.scholarpedia.org/article/Ensemble_learning) is restricted to different networks. Nowadays, it's typically used to indicate that you train multiple models and then combine them somehow, but I'm also not an expert in ensemble learning, to be honest. In any case, I've just looked at that chapter 9.1.6 again and he cites [399], which is a paper entitled ["When Networks Disagree: Ensemble Methods for Hybrid Neural Networks"](https://apps.dtic.mil/sti/pdfs/ADA260045.pdf).  
  > **username_1**: The book you're reading is not very new (although still a good book, as far as I'm concerned), so, at the time of writing that book, maybe "ensemble learning" was not very used yet as a standard term.  
  > **username_1**: Regarding the link above, I don't really know if it could help you. It talks about the correlation of errors in linear regression, so I thought it could be useful, at least, to understand what R. Rojas meant by "correlation". You could also interpret neural networks as performing non-linear regression, so that's another reason why I provided the link to that pdf, which, to be honest, I have only skimmed through.  
  > **username_2**: its a theoretical question about a practical approach. the payoff from this technique would tend to depend in part on the data, the network architecture, the features extracted, etc... in other words, the answer would mostly come from experiments in particular situations. basically there is some randomness in training also in which different nets may "find" or "emphasize" different actual features of the data.  
  > **username_0**: @username_2 Should I try to make the starting weights be far apart from network to network? something like randomly nudge weights so that they are different from the other networks' weights?  
  > **username_2**: yes try starting with different weights and possibly train the nets on different samples of the same data, ie try data segregation ideas. also Rojas was saying that it can work for as low as N=2 or N=3 but that doesnt mean to keep N in that range. the point is N varies for each situation and just test to find what works well/ is practical etc. you can ping me in [chat] or this chat room for further discussion https://chat.stackexchange.com/rooms/9446/theory-salon  
  
  ---
  
  ## 0 answers
  
  