<p>First of all, you can always construct an example to break assumptions.</p>
<p>&quot;<em>Say I take an MLP and generate data according to that MLP, then that MLP is the optimal solution, and anything you add on top of it worsens it, including naive regularization</em>&quot;</p>
<p>Going on your toy problem:</p>
<ol>
<li>there is a reason if the title says &quot;<em>for Image Recognition</em>&quot;</li>
<li>your counterexample works only for 1D mappings</li>
<li>The problem that the residual network helps with is OPTIMIZATION. In your example, no optimization has been done</li>
</ol>
<hr />
<p><em>Is X universally unilaterally unanimously better?</em><br />
Probably not. Let's all welcome the &quot;no free lunch theorem&quot;.</p>
<p><em>Is X probably going to help?</em><br />
We can talk about it</p>
<p><em>Are residual connections going to help you?</em><br />
If you have a deep network, most likely</p>

CC BY-SA 4.0
2024-09-06T16:26:25.103Z
In the paper itself they took an example of 1D mapping to prove their point and hence for I thought of another 1D case. Coming to the optimization part, I didn't ask the question for a particular problem driven approach and I need to get an optimal solution for that problem using a residual connection. I want to understand the residual connection itself not how its used for optimization.
ishaan
CC BY-SA 4.0
2024-09-08T08:36:29.360Z
@ishaan "I need to get an optimal solution" with a non-convex loss? good luck
Alberto
CC BY-SA 4.0
2024-09-05T19:09:02.397Z
2024-09-05T19:09:02.397Z
Alberto
<p>While reading the residual learning paper [<a href="https://arxiv.org/pdf/1512.03385" rel="nofollow noreferrer">1</a>], I found a problem to be quite unanswered. Suppose I am stacking a deep neural network to map an input to a output. Lets say a function H(x) does the required mapping. The function F(x) represents the residual to be learnt by the layers bypassing the input (x) only to be introduced/added before the output.</p>
<p>Consider a case where my H(x) maps any given instance of variable input x to 0 i.e. H(x) = 0 ∀x∈R. Now for this case F(x) = H(x) - x i.e. F(x) = -x ∀x∈R.</p>
<p>My question is doesn't learning F(x) = -x will be tougher for neural networks as compared to a simple mapping of H(x) = 0? Please explain.</p>

<deep-learning><computer-vision><activation-functions><residual-networks><skip-connections>
Is residual mapping always beneficial?
ishaan
# Is residual mapping always beneficial?

(Asked by: username_0 on 2024-09-05)

While reading the residual learning paper [[1](https://arxiv.org/pdf/1512.03385)], I found a problem to be quite unanswered. Suppose I am stacking a deep neural network to map an input to a output. Lets say a function H(x) does the required mapping. The function F(x) represents the residual to be learnt by the layers bypassing the input (x) only to be introduced/added before the output.

Consider a case where my H(x) maps any given instance of variable input x to 0 i.e. H(x) = 0 ∀x∈R. Now for this case F(x) = H(x) - x i.e. F(x) = -x ∀x∈R.

My question is doesn't learning F(x) = -x will be tougher for neural networks as compared to a simple mapping of H(x) = 0? Please explain.

**Tags:** deep-learning, computer-vision, activation-functions, residual-networks, skip-connections


---

## 1 answers

### Answer by username_1 on 2024-09-05. Score: 0

First of all, you can always construct an example to break assumptions.

"*Say I take an MLP and generate data according to that MLP, then that MLP is the optimal solution, and anything you add on top of it worsens it, including naive regularization*"

Going on your toy problem:

1. there is a reason if the title says "*for Image Recognition*"
2. your counterexample works only for 1D mappings
3. The problem that the residual network helps with is OPTIMIZATION. In your example, no optimization has been done

---

*Is X universally unilaterally unanimously better?*  
Probably not. Let's all welcome the "no free lunch theorem".

*Is X probably going to help?*  
We can talk about it

*Are residual connections going to help you?*  
If you have a deep network, most likely

> **Comments:**
> **username_0**: In the paper itself they took an example of 1D mapping to prove their point and hence for I thought of another 1D case. Coming to the optimization part, I didn't ask the question for a particular problem driven approach and I need to get an optimal solution for that problem using a residual connection. I want to understand the residual connection itself not how its used for optimization.  
> **username_1**: @username_0 "I need to get an optimal solution" with a non-convex loss? good luck  

---

