Answers.Body: <p>I think what is done in practice is that you just train multiple networks with different random seeds. This greatly reduces the possibility for getting bad models due to bad initialization.</p>
  <p>At the end you can just work with the model that shows the best performance on the validation set, or instead, use all the models as an ensemble.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-24T19:38:50.240Z
Answers.LastActivityDate: 2023-08-24T19:38:50.240Z
Answers.UserDisplayName: pi-tau
Body: <p>Now, the main idea behind ResNet is to enable the network to learn the residual value needed to modify the input for achieving the best result. In other words, given an input x, the output y is computed as y = F(x) + x, and the goal is to learn the function F (the filter). If the ideal output is the identity function, it implies that the model has discovered the best filter to be all zeros. This situation could arise due to bad initialization, resulting in the best filter being all zeros. Alternatively, it could be the optimal solution if there are no more residual blocks after it. If this is not the optimal solution, we can reinitialize the filters and retrain them.</p>
  <p>I would greatly appreciate any guidance or corrections.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-24T19:36:14.703Z
Comments.Text: I don't understand how could you tell if this is the optimal solution and leave it as it is; or if it is not the optimal solution and then re-init the weights ? It may be that one of the first res blocks learns to be an identity function instead of the last block. I mean, it does not have to be the last block.
Comments.UserDisplayName: pi-tau
Tags: <residual-networks>
Title: If a residual block learns to approximate the identity function, should we reinitialize the weights?
UserDisplayName: Legendary
ThreadText: # If a residual block learns to approximate the identity function, should we reinitialize the weights?
  
  (Asked by: username_0 on 2023-08-21)
  
  Now, the main idea behind ResNet is to enable the network to learn the residual value needed to modify the input for achieving the best result. In other words, given an input x, the output y is computed as y = F(x) + x, and the goal is to learn the function F (the filter). If the ideal output is the identity function, it implies that the model has discovered the best filter to be all zeros. This situation could arise due to bad initialization, resulting in the best filter being all zeros. Alternatively, it could be the optimal solution if there are no more residual blocks after it. If this is not the optimal solution, we can reinitialize the filters and retrain them.
  
  I would greatly appreciate any guidance or corrections.
  
  **Tags:** residual-networks
  
  > **Comments:**
  > **username_1**: I don't understand how could you tell if this is the optimal solution and leave it as it is; or if it is not the optimal solution and then re-init the weights ? It may be that one of the first res blocks learns to be an identity function instead of the last block. I mean, it does not have to be the last block.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-24. Score: 2
  
  I think what is done in practice is that you just train multiple networks with different random seeds. This greatly reduces the possibility for getting bad models due to bad initialization.
  
  At the end you can just work with the model that shows the best performance on the validation set, or instead, use all the models as an ensemble.
  
  
  ---
  
  