Answers.Body: <p>I am not sure what you are really looking for but I leave here <a href="https://arxiv.org/pdf/1905.11946.pdf" rel="nofollow noreferrer">this paper</a> here, where some intuition into that direction is provided. This paper compares the performance of a deep learning model scaling in 3 dimensions: resolution, width and depth. As depicted in their definition:
  <a href="https://i.sstatic.net/DqVpQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DqVpQ.png" alt="enter image description here" /></a></p>
  <p>If you go to section 3.2 you will see how scaling the different dimensions independently (resolution, width and depth) affects performance and how by performing a compound scaling the performance of the model is maximized (so there is a close relation). It is a very thorough ablation study. For me this was the paper where I understood how these Width, Depth and Resolution parameters come together.</p>
  <p>Rethinking Model Scaling for Convolutional Neural Network: <a href="https://arxiv.org/abs/1905.11946" rel="nofollow noreferrer">https://arxiv.org/abs/1905.11946</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-22T07:48:52.313Z
Answers.LastActivityDate: 2020-10-22T07:48:52.313Z
Answers.UserDisplayName: JVGD
Body: <p>All else being equal, including total neuron count, I give the following definitions:</p>
  <ul>
  <li>wide is a parallel ensemble, where good chunks of the neurons have the same inputs because the inputs are shared and they have different outputs.</li>
  <li>deep is a series ensemble, where for the most part neurons have as input the output of other neurons and few inputs are shared.</li>
  </ul>
  <p>For CART ensembles the parallel (wide) ensemble is a random forest while the series (deep) ensemble is a gradient boosted machine.  For several years the GBM was the <a href="https://www.kdnuggets.com/2016/03/xgboost-implementing-winningest-kaggle-algorithm-spark-flink.html" rel="nofollow noreferrer">&quot;winningest&quot; on kaggle</a>.</p>
  <p>Is there a parallel of that applied to Neural networks?  Is there some reasonable measure that indicates whether deep outperforms wide when it comes to neural networks?  If I had the same count of weights to throw at a tough problem, all else being equal should they be applied more strongly in parallel or in series?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-21T18:02:18.453Z
Comments.Text: Deep performs better because of multilevel feature extraction. At least the prevailing thought is that are hierarchy of features is implicitly extracted.
Comments.UserDisplayName: FourierFlux
Tags: <deep-neural-networks><gradient-boosting>
Title: Has "deep vs. wide" been resolved?
UserDisplayName: EngrStudent
ThreadText: # Has "deep vs. wide" been resolved?
  
  (Asked by: username_0 on 2020-10-21)
  
  All else being equal, including total neuron count, I give the following definitions:
  
  * wide is a parallel ensemble, where good chunks of the neurons have the same inputs because the inputs are shared and they have different outputs.
  * deep is a series ensemble, where for the most part neurons have as input the output of other neurons and few inputs are shared.
  
  For CART ensembles the parallel (wide) ensemble is a random forest while the series (deep) ensemble is a gradient boosted machine. For several years the GBM was the ["winningest" on kaggle](https://www.kdnuggets.com/2016/03/xgboost-implementing-winningest-kaggle-algorithm-spark-flink.html).
  
  Is there a parallel of that applied to Neural networks? Is there some reasonable measure that indicates whether deep outperforms wide when it comes to neural networks? If I had the same count of weights to throw at a tough problem, all else being equal should they be applied more strongly in parallel or in series?
  
  **Tags:** deep-neural-networks, gradient-boosting
  
  > **Comments:**
  > **username_1**: Deep performs better because of multilevel feature extraction. At least the prevailing thought is that are hierarchy of features is implicitly extracted.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-22. Score: 2
  
  I am not sure what you are really looking for but I leave here [this paper](https://arxiv.org/pdf/1905.11946.pdf) here, where some intuition into that direction is provided. This paper compares the performance of a deep learning model scaling in 3 dimensions: resolution, width and depth. As depicted in their definition:
  [![enter image description here](https://i.sstatic.net/DqVpQ.png)](https://i.sstatic.net/DqVpQ.png)
  
  If you go to section 3.2 you will see how scaling the different dimensions independently (resolution, width and depth) affects performance and how by performing a compound scaling the performance of the model is maximized (so there is a close relation). It is a very thorough ablation study. For me this was the paper where I understood how these Width, Depth and Resolution parameters come together.
  
  Rethinking Model Scaling for Convolutional Neural Network: <https://arxiv.org/abs/1905.11946>
  
  
  ---
  
  