Answers.Body: <p>It is not that we &quot;get more computing power&quot;, it is the fact that deep networks are more expressive than shallow ones, which is pretty much the result of what you have started stating about composition. It might be helpful to think of an example - here's a nice one I've written about <a href="https://github.com/Hadar933/Intro-to-Deep-Learning/blob/main/IDL_notes.pdf" rel="nofollow noreferrer">here</a></p>
  <blockquote>
  <p>We define the t-saw-tooth function as a piece-wise affine function
  with <span class="math-container">$t$</span> pieces. Also, we define the hat function as
  <span class="math-container">$hat(x)=relu(2relu(x)-4relu(x-0.5))$</span>.</p>
  <p>note that hat is a 4-saw-tooth function by definition. We can use hat
  to concatenate saw-tooth functions - let <span class="math-container">$t(x) =
   hat(x)+hat(x-1)$</span> comprises of two hats, and is a 6-saw-tooth function,
  and the composition <span class="math-container">$t(t(x))$</span> is a 10-saw-tooth, <span class="math-container">$t(t(t(x))))$</span> is a
  18-saw-tooth, and in general, a composition of <span class="math-container">$T$</span> functions (<span class="math-container">$T$</span>
  <span class="math-container">$t(x)$</span>'s) is a <span class="math-container">$(2+2^{T+1})$</span>-saw-tooth, namely, the composition is
  comprised of <span class="math-container">$2^T$</span> hats. How can we represent this composition using a
  shallow network? remember that every neuron in the (single) hidden
  layer &quot;represents&quot; a single relu, therefore two neurons can represent
  a single hat. As there are exponentially many hats, a shallow network
  will have to use exponentially many neurons (<span class="math-container">$2^{T+1}$</span> to be exact).
  On the other hand, a deep neural network would only need <span class="math-container">$\Theta(T)$</span>
  (if, for example, we use <span class="math-container">$T$</span> layers and <span class="math-container">$2$</span> nodes per layer). This is
  the case because the deep network <span class="math-container">$i'th$</span> layer receives as input the
  values of the <span class="math-container">$(i-1)'th$</span> layer, and for that matter is represents the
  composition of <span class="math-container">$t(x)$</span> simply as a product of its architectural design.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-31T20:08:43.807Z
Answers.LastActivityDate: 2022-09-05T10:14:57.287Z
Answers.LastEditDate: 2022-09-05T10:14:57.287Z
Answers.UserDisplayName: Hadar Sharvit
Answers.Body: <p>It would be simple, intuitive answer but generally deeper networks have more &quot;space&quot; to learn more complex features. They start from very simple &quot;shapes&quot; (using CNNs as an example) and gradually build towards more complex ones. Having more layers means there can be a lot of intermediary   stages that will help in constituting final features (near the end of the network). This assures that the final features can be complex. In that case more details are considered in the middle layers and last layers can take advantage of these details.</p>
  <p>Whereas shallow networks don't have that space for developing complicated features at the end. They must make use of only few layers therefore the increase of features details will be big between layers, making it harder for complex &quot;shapes&quot; to be found at the end. A lot of useful information maybe lost in the process.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-31T20:59:33.810Z
Answers.LastActivityDate: 2022-08-31T20:59:33.810Z
Answers.UserDisplayName: GKozinski
Answers.Body: <p>I agree that in practice having deeper networks makes it simpler to construct more expressive functions for the reasons that other people have mentioned. However, while it is a practical convenience, it is not a theoretically necessity. The Wikipedia page on the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">Universal approximation theorem</a> reminds us that</p>
  <blockquote>
  <p>Kurt Hornik, Maxwell Stinchcombe, and Halbert White showed in 1989 that multilayer feed-forward networks with as few as one hidden layer are universal approximators.</p>
  </blockquote>
  <p>That is to say, <em>in line of principle</em> even a network with a single hidden layer can approximate any function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-22T18:39:44.080Z
Answers.LastActivityDate: 2022-09-22T18:39:44.080Z
Answers.UserDisplayName: Giorgio
Answers.Body: <p>Let's try to get the intuition with an example.</p>
  <p>If you're already familiar with CNNs, we can use the example of <a href="https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks" rel="nofollow noreferrer">Feature Maps</a>. Feature Maps help us visualize what CNNs learn at each layer. The observation is that the ones in the first layer capture the fine details of the input image, the ones in the second use them to get the finer details (that's why the feature map is less visualizable) and the same goes on as we go deeper. The feature maps at the second last layer may be hardly visualizable but has more information than the previous layers to predict the output.</p>
  <p>In case you're not familiar with CNNs, we can use the example of a multi-layered artificial neural network. The idea is same as the CNN. The first layer captures the simple details of the input and creates some features, the second layer uses those features to get the better details and the same goes on till the second last layer. We feed the final features to an output function which gives us the final output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-04T09:03:06.783Z
Answers.LastActivityDate: 2022-09-04T09:03:06.783Z
Answers.UserDisplayName: babkr
Body: <p>I know by the expressiveness of a neural networks that it can be seen as a chain of function compositions, i.e. <span class="math-container">$g(f(.. z(x)..))$</span> and also that, if we go deep, we can approximate complex functions <span class="math-container">$f: \mathbb{R} \rightarrow [0,1]$</span> with a lower number of units.</p>
  <p>But why, if we go deeper, do we get computing power?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-31T17:28:44.320Z
Comments.Text: You have to explain what do you exactly mean by "getting computing power".
Comments.UserDisplayName: GKozinski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-31T17:38:40.617Z
Comments.Text: Thaks, I edited, hope is much clear now.
Comments.UserDisplayName: f_s
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-31T19:50:37.583Z
Comments.Text: What do you mean by computing power? I understand computing power as the ability for a computing to do more operations by seconds, which is clearly not what you mean.
Comments.UserDisplayName: Raphael Lopez Kaufman
Tags: <deep-neural-networks>
Title: Why, in deep learning, do we get computational power by going deeper?
UserDisplayName: f_s
ThreadText: # Why, in deep learning, do we get computational power by going deeper?
  
  (Asked by: username_0 on 2022-08-31)
  
  I know by the expressiveness of a neural networks that it can be seen as a chain of function compositions, i.e. $g(f(.. z(x)..))$ and also that, if we go deep, we can approximate complex functions $f: \mathbb{R} \rightarrow [0,1]$ with a lower number of units.
  
  But why, if we go deeper, do we get computing power?
  
  **Tags:** deep-neural-networks
  > **Comments:**
  > **username_1**: You have to explain what do you exactly mean by "getting computing power".  
  > **username_0**: Thaks, I edited, hope is much clear now.  
  > **username_2**: What do you mean by computing power? I understand computing power as the ability for a computing to do more operations by seconds, which is clearly not what you mean.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_3 on 2022-08-31. Score: 3
  
  It is not that we "get more computing power", it is the fact that deep networks are more expressive than shallow ones, which is pretty much the result of what you have started stating about composition. It might be helpful to think of an example - here's a nice one I've written about [here](https://github.com/Hadar933/Intro-to-Deep-Learning/blob/main/IDL_notes.pdf)
  
  > We define the t-saw-tooth function as a piece-wise affine function
  > with $t$ pieces. Also, we define the hat function as
  > $hat(x)=relu(2relu(x)-4relu(x-0.5))$.
  >
  > note that hat is a 4-saw-tooth function by definition. We can use hat
  > to concatenate saw-tooth functions - let $t(x) =
  > hat(x)+hat(x-1)$ comprises of two hats, and is a 6-saw-tooth function,
  > and the composition $t(t(x))$ is a 10-saw-tooth, $t(t(t(x))))$ is a
  > 18-saw-tooth, and in general, a composition of $T$ functions ($T$
  > $t(x)$'s) is a $(2+2^{T+1})$-saw-tooth, namely, the composition is
  > comprised of $2^T$ hats. How can we represent this composition using a
  > shallow network? remember that every neuron in the (single) hidden
  > layer "represents" a single relu, therefore two neurons can represent
  > a single hat. As there are exponentially many hats, a shallow network
  > will have to use exponentially many neurons ($2^{T+1}$ to be exact).
  > On the other hand, a deep neural network would only need $\Theta(T)$
  > (if, for example, we use $T$ layers and $2$ nodes per layer). This is
  > the case because the deep network $i'th$ layer receives as input the
  > values of the $(i-1)'th$ layer, and for that matter is represents the
  > composition of $t(x)$ simply as a product of its architectural design.
  
  
  ---
  
  ### Answer by username_1 on 2022-08-31. Score: 2
  
  It would be simple, intuitive answer but generally deeper networks have more "space" to learn more complex features. They start from very simple "shapes" (using CNNs as an example) and gradually build towards more complex ones. Having more layers means there can be a lot of intermediary stages that will help in constituting final features (near the end of the network). This assures that the final features can be complex. In that case more details are considered in the middle layers and last layers can take advantage of these details.
  
  Whereas shallow networks don't have that space for developing complicated features at the end. They must make use of only few layers therefore the increase of features details will be big between layers, making it harder for complex "shapes" to be found at the end. A lot of useful information maybe lost in the process.
  
  
  ---
  
  ### Answer by username_4 on 2022-09-22. Score: 1
  
  I agree that in practice having deeper networks makes it simpler to construct more expressive functions for the reasons that other people have mentioned. However, while it is a practical convenience, it is not a theoretically necessity. The Wikipedia page on the [Universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) reminds us that
  
  > Kurt Hornik, Maxwell Stinchcombe, and Halbert White showed in 1989 that multilayer feed-forward networks with as few as one hidden layer are universal approximators.
  
  That is to say, *in line of principle* even a network with a single hidden layer can approximate any function.
  
  
  ---
  
  ### Answer by username_5 on 2022-09-04. Score: 1
  
  Let's try to get the intuition with an example.
  
  If you're already familiar with CNNs, we can use the example of [Feature Maps](https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks). Feature Maps help us visualize what CNNs learn at each layer. The observation is that the ones in the first layer capture the fine details of the input image, the ones in the second use them to get the finer details (that's why the feature map is less visualizable) and the same goes on as we go deeper. The feature maps at the second last layer may be hardly visualizable but has more information than the previous layers to predict the output.
  
  In case you're not familiar with CNNs, we can use the example of a multi-layered artificial neural network. The idea is same as the CNN. The first layer captures the simple details of the input and creates some features, the second layer uses those features to get the better details and the same goes on till the second last layer. We feed the final features to an output function which gives us the final output.
  
  
  ---
  
  