Answers.Body: <p>Random residual networks for many non-linearities such as tanh live on the edge of chaos,
  in that the cosine distance of two input vectors will converge to a fixed point at a polynomial rate, rather than an exponential rate, as with vanilla tanh networks. Thus a typical residual network will slowly cross the stable-chaotic boundary with depth, hovering around this boundary for many layers. Basically it does not “forget” the geometry of the input space “very quickly”. So even if we make them considerably deep, they work better the vanilla networks. </p>
  
  <p>For more information on the propagation of information in residual networks - <a href="https://ai.google/research/pubs/pub46492/" rel="nofollow noreferrer">Mean Field Residual Networks: On the Edge of Chaos</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-14T14:04:44.453Z
Answers.LastActivityDate: 2019-10-14T14:04:44.453Z
Answers.UserDisplayName: Snehal Reddy
Answers.Body: <p>Imagine a genie grants you three wishes. Because you are an ambitious deep learning researcher your first wish is a perfect solution for a 1000-layer NN for Image Net, which promptly appears on your laptop.</p>
  
  <p>Now a genie induced solution doesn't give you any intuition how it might be interpreted as an ensemble, but do you really believe that you need 1000 layers of abstraction to distinguish a cat from a dog? As the authors of the "ensemble paper" mention themselves, this is definitely not true for biological systems.</p>
  
  <p>Of course you could waste your second wish on a decomposition of the solution into an ensemble of networks, and I'm pretty sure the genie would be able to oblige. The reason being that part of the power of a deep network will always come from the ensemble effect.</p>
  
  <p>So it is not surprising that two very successful tricks to train deep networks, dropout and residual networks, have an immediate interpretation as implicit ensemble. Therefore "it's not depth, but the ensemble" strikes me as a false dichotomy. You would really only say that if you honestly believed that you need hundreds or thousands of levels of abstraction to classify images with human accuracy. </p>
  
  <p>I suggest you use the last wish for something else, maybe a pinacolada. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-20T14:40:30.907Z
Answers.LastActivityDate: 2016-09-20T14:40:30.907Z
Answers.UserDisplayName: BlindKungFuMaster
Body: <p>The question is about the architecture of Deep Residual Networks (<strong>ResNets</strong>). The model that won the 1-st places at <a href="http://image-net.org/challenges/LSVRC/2015/results" rel="nofollow noreferrer">"Large Scale Visual Recognition Challenge 2015" (ILSVRC2015)</a> in all five main tracks:</p>
  
  <blockquote>
    <ul>
    <li><em>ImageNet Classification: “Ultra-deep” (quote Yann) 152-layer nets</em> </li>
    <li><em>ImageNet Detection: 16% better than 2nd</em></li>
    <li><em>ImageNet Localization: 27% better than 2nd</em></li>
    <li><em>COCO Detection: 11% better than 2nd</em></li>
    <li><em>COCO Segmentation: 12% better than 2nd<br><br></em>
    <em>Source:</em> <a href="http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf" rel="nofollow noreferrer"><em>MSRA @ ILSVRC &amp; COCO 2015 competitions (presentation, 2-nd slide)</em></a></li>
    </ul>
  </blockquote>
  
  <p>This work is described in the following article:</p>
  
  <blockquote>
    <p><a href="http://arxiv.org/abs/1512.03385" rel="nofollow noreferrer"><em>Deep Residual Learning for Image Recognition (2015, PDF)</em></a></p>
  </blockquote>
  
  <hr>
  
  <p><strong>Microsoft Research team</strong> (developers of ResNets: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun) in their article:</p>
  
  <blockquote>
    <p><a href="https://arxiv.org/pdf/1603.05027.pdf" rel="nofollow noreferrer">"<em>Identity Mappings in Deep Residual Networks (2016)</em>"</a></p>
  </blockquote>
  
  <p>state that <strong>depth</strong> plays a key role:</p>
  
  <blockquote>
    <p><em>"<strong>We obtain these results via a simple but essential concept — going deeper. These results demonstrate the potential of pushing the limits of depth.</strong>"</em></p>
  </blockquote>
  
  <p>It is emphasized in their <a href="http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf" rel="nofollow noreferrer">presentation</a> also (deeper - better):<br> </p>
  
  <blockquote>
    <p><em>- "A deeper model should not have higher training error."<br> 
    - "Deeper ResNets have lower training error, and also lower test error."<br> 
    - "Deeper ResNets have lower error."<br>
    - "All benefit more from deeper features – cumulative gains!"<br>
    - "Deeper is still better."</em></p>
  </blockquote>
  
  <p>Here is the sctructure of 34-layer residual (for reference):
  <a href="https://i.sstatic.net/L8m0X.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/L8m0X.png" alt="enter image description here"></a></p>
  
  <hr>
  
  <p>But recently I have found one theory that introduces a novel interpretation of residual networks showing they are exponential ensembles:</p>
  
  <blockquote>
    <p><a href="https://arxiv.org/abs/1605.06431" rel="nofollow noreferrer"><em>Residual Networks are Exponential Ensembles of Relatively Shallow Networks (2016)</em></a></p>
  </blockquote>
  
  <p>Deep Resnets are described as many shallow networks whose outputs are pooled at various depths. 
  There is a picture in the article. I attach it with explanation:</p>
  
  <blockquote>
    <p><a href="https://i.sstatic.net/PGhK2.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PGhK2.jpg" alt="enter image description here"></a> Residual Networks are
    conventionally shown as (a), which is a natural representation of
    Equation (1). When we expand this formulation to Equation (6), we
    obtain an unraveled view of a 3-block residual network (b). From this
    view, it is apparent that residual networks have O(2^n) implicit paths
    connecting input and output and that adding a block doubles the number
    of paths.</p>
  </blockquote>
  
  <p>In conclusion of the article it is stated:</p>
  
  <blockquote>
    <p><strong>It is not depth, but the ensemble that makes residual networks strong</strong>.
    Residual networks push the limits of network multiplicity, not network
    depth. Our proposed unraveled view and the lesion study show that
    residual networks are an implicit ensemble of exponentially many
    networks. If most of the paths that contribute gradient are very short
    compared to the overall depth of the network, <strong>increased depth</strong>
    alone <strong>can’t be the key characteristic</strong> of residual networks. We now
    believe that <strong>multiplicity</strong>, the network’s expressability in the
    terms of the number of paths, plays <strong>a key role</strong>.</p>
  </blockquote>
  
  <p>But it is only a recent theory that can be confirmed or refuted. It happens sometimes that some theories are refuted and articles are withdrawn.</p>
  
  <hr>
  
  <p>Should we think of deep ResNets as an ensemble after all? <strong>Ensemble</strong> or <strong>depth</strong> makes residual networks so strong? Is it possible that even the developers themselves do not quite perceive what their own model represents and what is the key concept in it?</p>
  
Tags: <neural-networks><machine-learning><deep-learning><deep-neural-networks><residual-networks>
Title: Should deep residual networks be viewed as an ensemble of networks?
UserDisplayName: Erba Aitbayev
ThreadText: # Should deep residual networks be viewed as an ensemble of networks?
  
  (Asked by: username_0 on 2016-09-20)
  
  The question is about the architecture of Deep Residual Networks (**ResNets**). The model that won the 1-st places at ["Large Scale Visual Recognition Challenge 2015" (ILSVRC2015)](http://image-net.org/challenges/LSVRC/2015/results) in all five main tracks:
  
  > * *ImageNet Classification: “Ultra-deep” (quote Yann) 152-layer nets*
  > * *ImageNet Detection: 16% better than 2nd*
  > * *ImageNet Localization: 27% better than 2nd*
  > * *COCO Detection: 11% better than 2nd*
  > * *COCO Segmentation: 12% better than 2nd*
  >   *Source:* [*MSRA @ ILSVRC & COCO 2015 competitions (presentation, 2-nd slide)*](http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf)
  
  This work is described in the following article:
  
  > [*Deep Residual Learning for Image Recognition (2015, PDF)*](http://arxiv.org/abs/1512.03385)
  
  ---
  
  **Microsoft Research team** (developers of ResNets: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun) in their article:
  
  > ["*Identity Mappings in Deep Residual Networks (2016)*"](https://arxiv.org/pdf/1603.05027.pdf)
  
  state that **depth** plays a key role:
  
  > *"**We obtain these results via a simple but essential concept — going deeper. These results demonstrate the potential of pushing the limits of depth.**"*
  
  It is emphasized in their [presentation](http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf) also (deeper - better):
  
  > *- "A deeper model should not have higher training error."  
  > - "Deeper ResNets have lower training error, and also lower test error."  
  > - "Deeper ResNets have lower error."  
  > - "All benefit more from deeper features – cumulative gains!"  
  > - "Deeper is still better."*
  
  Here is the sctructure of 34-layer residual (for reference):
  [![enter image description here](https://i.sstatic.net/L8m0X.png)](https://i.sstatic.net/L8m0X.png)
  
  ---
  
  But recently I have found one theory that introduces a novel interpretation of residual networks showing they are exponential ensembles:
  
  > [*Residual Networks are Exponential Ensembles of Relatively Shallow Networks (2016)*](https://arxiv.org/abs/1605.06431)
  
  Deep Resnets are described as many shallow networks whose outputs are pooled at various depths.
  There is a picture in the article. I attach it with explanation:
  
  > [![enter image description here](https://i.sstatic.net/PGhK2.jpg)](https://i.sstatic.net/PGhK2.jpg) Residual Networks are
  > conventionally shown as (a), which is a natural representation of
  > Equation (1). When we expand this formulation to Equation (6), we
  > obtain an unraveled view of a 3-block residual network (b). From this
  > view, it is apparent that residual networks have O(2^n) implicit paths
  > connecting input and output and that adding a block doubles the number
  > of paths.
  
  In conclusion of the article it is stated:
  
  > **It is not depth, but the ensemble that makes residual networks strong**.
  > Residual networks push the limits of network multiplicity, not network
  > depth. Our proposed unraveled view and the lesion study show that
  > residual networks are an implicit ensemble of exponentially many
  > networks. If most of the paths that contribute gradient are very short
  > compared to the overall depth of the network, **increased depth**
  > alone **can’t be the key characteristic** of residual networks. We now
  > believe that **multiplicity**, the network’s expressability in the
  > terms of the number of paths, plays **a key role**.
  
  But it is only a recent theory that can be confirmed or refuted. It happens sometimes that some theories are refuted and articles are withdrawn.
  
  ---
  
  Should we think of deep ResNets as an ensemble after all? **Ensemble** or **depth** makes residual networks so strong? Is it possible that even the developers themselves do not quite perceive what their own model represents and what is the key concept in it?
  
  **Tags:** neural-networks, machine-learning, deep-learning, deep-neural-networks, residual-networks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2016-09-20. Score: 6 | Accepted answer
  
  Imagine a genie grants you three wishes. Because you are an ambitious deep learning researcher your first wish is a perfect solution for a 1000-layer NN for Image Net, which promptly appears on your laptop.
  
  Now a genie induced solution doesn't give you any intuition how it might be interpreted as an ensemble, but do you really believe that you need 1000 layers of abstraction to distinguish a cat from a dog? As the authors of the "ensemble paper" mention themselves, this is definitely not true for biological systems.
  
  Of course you could waste your second wish on a decomposition of the solution into an ensemble of networks, and I'm pretty sure the genie would be able to oblige. The reason being that part of the power of a deep network will always come from the ensemble effect.
  
  So it is not surprising that two very successful tricks to train deep networks, dropout and residual networks, have an immediate interpretation as implicit ensemble. Therefore "it's not depth, but the ensemble" strikes me as a false dichotomy. You would really only say that if you honestly believed that you need hundreds or thousands of levels of abstraction to classify images with human accuracy.
  
  I suggest you use the last wish for something else, maybe a pinacolada.
  
  
  ---
  
  ### Answer by username_1 on 2019-10-14. Score: 1
  
  Random residual networks for many non-linearities such as tanh live on the edge of chaos,
  in that the cosine distance of two input vectors will converge to a fixed point at a polynomial rate, rather than an exponential rate, as with vanilla tanh networks. Thus a typical residual network will slowly cross the stable-chaotic boundary with depth, hovering around this boundary for many layers. Basically it does not “forget” the geometry of the input space “very quickly”. So even if we make them considerably deep, they work better the vanilla networks.
  
  For more information on the propagation of information in residual networks - [Mean Field Residual Networks: On the Edge of Chaos](https://ai.google/research/pubs/pub46492/)
  
  
  ---
  
  