Answers.Body: <p>First of all, you can always construct an example to break assumptions.</p>
  <p>&quot;<em>Say I take an MLP and generate data according to that MLP, then that MLP is the optimal solution, and anything you add on top of it worsens it, including naive regularization</em>&quot;</p>
  <p>Going on your toy problem:</p>
  <ol>
  <li>there is a reason if the title says &quot;<em>for Image Recognition</em>&quot;</li>
  <li>your counterexample works only for 1D mappings</li>
  <li>The problem that the residual network helps with is OPTIMIZATION. In your example, no optimization has been done</li>
  </ol>
  <hr />
  <p><em>Is X universally unilaterally unanimously better?</em><br />
  Probably not. Let's all welcome the &quot;no free lunch theorem&quot;.</p>
  <p><em>Is X probably going to help?</em><br />
  We can talk about it</p>
  <p><em>Are residual connections going to help you?</em><br />
  If you have a deep network, most likely</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-06T16:26:25.103Z
Answers.Text: In the paper itself they took an example of 1D mapping to prove their point and hence for I thought of another 1D case. Coming to the optimization part, I didn't ask the question for a particular problem driven approach and I need to get an optimal solution for that problem using a residual connection. I want to understand the residual connection itself not how its used for optimization.
Answers.UserDisplayName: ishaan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-08T08:36:29.360Z
Answers.Text: @ishaan "I need to get an optimal solution" with a non-convex loss? good luck
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-05T19:09:02.397Z
Answers.LastActivityDate: 2024-09-05T19:09:02.397Z
Answers.UserDisplayName: Alberto
Body: <p>While reading the residual learning paper [<a href="https://arxiv.org/pdf/1512.03385" rel="nofollow noreferrer">1</a>], I found a problem to be quite unanswered. Suppose I am stacking a deep neural network to map an input to a output. Lets say a function H(x) does the required mapping. The function F(x) represents the residual to be learnt by the layers bypassing the input (x) only to be introduced/added before the output.</p>
  <p>Consider a case where my H(x) maps any given instance of variable input x to 0 i.e. H(x) = 0 ∀x∈R. Now for this case F(x) = H(x) - x i.e. F(x) = -x ∀x∈R.</p>
  <p>My question is doesn't learning F(x) = -x will be tougher for neural networks as compared to a simple mapping of H(x) = 0? Please explain.</p>
  
Tags: <deep-learning><computer-vision><activation-functions><residual-networks><skip-connections>
Title: Is residual mapping always beneficial?
UserDisplayName: ishaan
ThreadText: # Is residual mapping always beneficial?
  
  (Asked by: username_0 on 2024-09-05)
  
  While reading the residual learning paper [[1](https://arxiv.org/pdf/1512.03385)], I found a problem to be quite unanswered. Suppose I am stacking a deep neural network to map an input to a output. Lets say a function H(x) does the required mapping. The function F(x) represents the residual to be learnt by the layers bypassing the input (x) only to be introduced/added before the output.
  
  Consider a case where my H(x) maps any given instance of variable input x to 0 i.e. H(x) = 0 ∀x∈R. Now for this case F(x) = H(x) - x i.e. F(x) = -x ∀x∈R.
  
  My question is doesn't learning F(x) = -x will be tougher for neural networks as compared to a simple mapping of H(x) = 0? Please explain.
  
  **Tags:** deep-learning, computer-vision, activation-functions, residual-networks, skip-connections
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-05. Score: 0
  
  First of all, you can always construct an example to break assumptions.
  
  "*Say I take an MLP and generate data according to that MLP, then that MLP is the optimal solution, and anything you add on top of it worsens it, including naive regularization*"
  
  Going on your toy problem:
  
  1. there is a reason if the title says "*for Image Recognition*"
  2. your counterexample works only for 1D mappings
  3. The problem that the residual network helps with is OPTIMIZATION. In your example, no optimization has been done
  
  ---
  
  *Is X universally unilaterally unanimously better?*  
  Probably not. Let's all welcome the "no free lunch theorem".
  
  *Is X probably going to help?*  
  We can talk about it
  
  *Are residual connections going to help you?*  
  If you have a deep network, most likely
  
  > **Comments:**
  > **username_0**: In the paper itself they took an example of 1D mapping to prove their point and hence for I thought of another 1D case. Coming to the optimization part, I didn't ask the question for a particular problem driven approach and I need to get an optimal solution for that problem using a residual connection. I want to understand the residual connection itself not how its used for optimization.  
  > **username_1**: @ishaan "I need to get an optimal solution" with a non-convex loss? good luck  
  
  ---
  
  