Answers.Body: <p>Suppose that in the general case there is a way to convert a 3 layer network in a 2 layer network in an exact/closed form way</p>
  <p>Then, given a <span class="math-container">$N$</span> layer network, considering the first <span class="math-container">$N-3$</span> layers as just a mere preprocessing, you can squeeze the last <span class="math-container">$3$</span> in <span class="math-container">$2$</span> layer, thus having a final network of <span class="math-container">$N-1$</span> layers</p>
  <p>Now, rinse and repeat, and you have magically transformed a <span class="math-container">$N$</span> layers network in a <span class="math-container">$2$</span> layers network</p>
  <p>Clearly there isn't such a way, mainly due to the non linearity in between layers (in the linear case, you might be able to do so, but you have not given any specification/hypothesis on the layers structure)</p>
  <p>Having said this, there are ways to approximate this, indeed a <span class="math-container">$3$</span> layer network is still a function, and thus you can use a <span class="math-container">$2$</span> layers function approximator to approximate it, and this is usually called distillation, as <a href="https://ai.stackexchange.com/users/36821/david">@David</a> pointed out in the comments, however this is everything but guaranteed to converge (if you don't sample enough points, and if the smaller network has not enough capacity)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-23T11:07:32.850Z
Answers.LastActivityDate: 2024-03-23T11:07:32.850Z
Answers.UserDisplayName: Alberto
Body: <p>Currently, my professor asked me to find an explicit formula to convert 3 layer network to 2 layer network. I've read some paper about the general properties of neural network, how its complexity is related to depth and width, but I haven't found a single paper talking about the conversion of network with different depths. Is there any paper related to this subject?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-23T09:09:15.133Z
Comments.Text: They could be expecting you to use distillation. That would be to train the smaller network to make the same predictions as the larger network.
Comments.UserDisplayName: David
Tags: <neural-networks>
Title: Is there a way to convert 3 layer network to 2 layer network with specific math formula?
UserDisplayName: Xiaoyang Li
ThreadText: # Is there a way to convert 3 layer network to 2 layer network with specific math formula?
  
  (Asked by: username_0 on 2024-03-23)
  
  Currently, my professor asked me to find an explicit formula to convert 3 layer network to 2 layer network. I've read some paper about the general properties of neural network, how its complexity is related to depth and width, but I haven't found a single paper talking about the conversion of network with different depths. Is there any paper related to this subject?
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: They could be expecting you to use distillation. That would be to train the smaller network to make the same predictions as the larger network.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-03-23. Score: 1
  
  Suppose that in the general case there is a way to convert a 3 layer network in a 2 layer network in an exact/closed form way
  
  Then, given a $N$ layer network, considering the first $N-3$ layers as just a mere preprocessing, you can squeeze the last $3$ in $2$ layer, thus having a final network of $N-1$ layers
  
  Now, rinse and repeat, and you have magically transformed a $N$ layers network in a $2$ layers network
  
  Clearly there isn't such a way, mainly due to the non linearity in between layers (in the linear case, you might be able to do so, but you have not given any specification/hypothesis on the layers structure)
  
  Having said this, there are ways to approximate this, indeed a $3$ layer network is still a function, and thus you can use a $2$ layers function approximator to approximate it, and this is usually called distillation, as [@David](https://ai.stackexchange.com/users/36821/david) pointed out in the comments, however this is everything but guaranteed to converge (if you don't sample enough points, and if the smaller network has not enough capacity)
  
  
  ---
  
  