Answers.Body: <p>Isn't this just a very short recurrent neural network? Same issues apply, although they are less severe since you aren't applying as many recurrent iterations. Once you start &quot;nesting&quot; them more, most typical issues are vanishing and exploding gradients.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T17:44:19.587Z
Answers.Text: Thank you for the insightful answer. Does gradients vanishing/exploding also become severe if the input dimension increases while the recurrent iterations stay the same?
Answers.UserDisplayName: Zuba Tupaki
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-15T11:35:40.560Z
Answers.Text: To my knowledge the input dimension doesn't affect the math on how stacked model's gradients tend to either zero or infinity (depending whether the multiplier is less or greater than one).
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T14:15:09.260Z
Answers.LastActivityDate: 2023-11-13T14:15:09.260Z
Answers.UserDisplayName: NikoNyrh
Body: <p>everyone. I am working on a nested neural network architecture. For the sake of better understanding my question, simply assume the loss is</p>
  <p><span class="math-container">$L = G(k’) - H(k'')$</span></p>
  <p>where <span class="math-container">$G$</span> and <span class="math-container">$H$</span> are two functions we do not need to know but variable <span class="math-container">$k'$</span> and <span class="math-container">$k''$</span> stem from neural network NNs such that</p>
  <p><span class="math-container">$k’ = NNs(k),\quad k’’ = NNs(k’)$</span></p>
  <p>so this would make the acquirement of <span class="math-container">$k''$</span> go through a nested architecture of the same neural networks, such that</p>
  <p><span class="math-container">$k'' = NNs(NNs(k))$</span></p>
  <p>Apparently, to obtain output <span class="math-container">$k’’$</span> the same neural network gets nested once:</p>
  <p><span class="math-container">$k \to NNs \to k′ \to NNs \to k′′$</span></p>
  <p>So my question is what is the potential issue of the nested neural network? Is there any paper that I could look into? Pytorch forum told that stale forward activations are used during a backward pass could raise issue, but I cannot find any example (the same question has been asked at <a href="https://discuss.pytorch.org/t/how-does-backward-propagation-work-in-a-nested-neural-network/189962" rel="nofollow noreferrer">Pytorch forum</a>).</p>
  
Tags: <neural-networks><machine-learning><deep-learning><deep-neural-networks>
Title: What is the potential issue of nested neural networks
UserDisplayName: Zuba Tupaki
ThreadText: # What is the potential issue of nested neural networks
  
  (Asked by: username_0 on 2023-11-13)
  
  everyone. I am working on a nested neural network architecture. For the sake of better understanding my question, simply assume the loss is
  
  $L = G(k’) - H(k'')$
  
  where $G$ and $H$ are two functions we do not need to know but variable $k'$ and $k''$ stem from neural network NNs such that
  
  $k’ = NNs(k),\quad k’’ = NNs(k’)$
  
  so this would make the acquirement of $k''$ go through a nested architecture of the same neural networks, such that
  
  $k'' = NNs(NNs(k))$
  
  Apparently, to obtain output $k’’$ the same neural network gets nested once:
  
  $k \to NNs \to k′ \to NNs \to k′′$
  
  So my question is what is the potential issue of the nested neural network? Is there any paper that I could look into? Pytorch forum told that stale forward activations are used during a backward pass could raise issue, but I cannot find any example (the same question has been asked at [Pytorch forum](https://discuss.pytorch.org/t/how-does-backward-propagation-work-in-a-nested-neural-network/189962)).
  
  **Tags:** neural-networks, machine-learning, deep-learning, deep-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-13. Score: 1 | Accepted answer
  
  Isn't this just a very short recurrent neural network? Same issues apply, although they are less severe since you aren't applying as many recurrent iterations. Once you start "nesting" them more, most typical issues are vanishing and exploding gradients.
  
  > **Comments:**
  > **username_0**: Thank you for the insightful answer. Does gradients vanishing/exploding also become severe if the input dimension increases while the recurrent iterations stay the same?  
  > **username_1**: To my knowledge the input dimension doesn't affect the math on how stacked model's gradients tend to either zero or infinity (depending whether the multiplier is less or greater than one).  
  
  ---
  
  