Body: <p>Consider the following neural network with <span class="math-container">$\ell$</span> layers.</p>
  <p><span class="math-container">$$i_0 \rightarrow h_1  \rightarrow h_2 \rightarrow h_3 \cdots \rightarrow h_{\ell-1}  \rightarrow o_{\ell} ,$$</span></p>
  <p>where <span class="math-container">$i, h, o$</span> stands for input, hidden and output layer respectively.</p>
  <p>In general, an input passes from <span class="math-container">$i_0$</span> to <span class="math-container">$o_{\ell}$</span>, which is known as the <em>forward pass</em>. And then the weight updating happens from <span class="math-container">$o_{\ell}$</span>  to <span class="math-container">$i_0$</span> which is called <em>backward pass</em>.</p>
  <p>I want to know whether the following mechanism exists in literature assuming that all layers have the same input and output dimensions.</p>
  <p>For each iteration</p>
  <ol>
  <li>Select a subset <span class="math-container">$L  \subseteq \{0, 1, 2, 3, \cdots, \ell\}$</span> randomly.</li>
  <li>Input passes through layers whose indices are present in <span class="math-container">$L$</span> only i.e, forward pass happens by dropping some layers.</li>
  <li>Update weights for layers whose indices are in <span class="math-container">$L$</span> i.e., update the weights of layers which are participated in step (2).</li>
  </ol>
  <p>What is the name of the technique mentioned above, if it is present in literature?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T07:21:56.790Z
Comments.Text: This should be relatively easy to put together in PyTorch or Chainer (any lib that can dope with dynamic graphs - doing this is in e.g. Tensorflow would be much harder). Have you tried it, does it have any useful impact? I'd imagine it may work as a form of regularisation. It also has *some* relationship to unrolled RNNs, as each layer will tend to work as if it is processing the same kind of input. However, the technique is unlikely to have any references or have been given a name if the results are not good.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T12:29:37.337Z
Comments.Text: The closest technique to what you're proposing that comes to my mind right now is _dropout_, but, in dropout, we _randomly_ drop some neurons from the neural network, so not full layers.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T14:37:20.543Z
Comments.Text: @NeilSlater I didn't try yet. Need to do.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T16:36:38.330Z
Comments.Text: With some playing around, you could probably find some scenarios where this benefits the learning of a model. Unfortunately, I don't think this method will ever outperform dropout as it simply drops too much at once. No definite answer here though, you'll need to do your own testing to evaluate it's effectiveness
Comments.UserDisplayName: Recessive
Tags: <neural-networks><training><reference-request><forward-pass>
Title: Is there any existing mechanism that allows us to pass input from randomly selected layers of neural network per iteration?
UserDisplayName: hanugm
ThreadText: # Is there any existing mechanism that allows us to pass input from randomly selected layers of neural network per iteration?
  
  (Asked by: username_0 on 2021-07-26)
  
  Consider the following neural network with $\ell$ layers.
  
  $$i\_0 \rightarrow h\_1 \rightarrow h\_2 \rightarrow h\_3 \cdots \rightarrow h\_{\ell-1} \rightarrow o\_{\ell} ,$$
  
  where $i, h, o$ stands for input, hidden and output layer respectively.
  
  In general, an input passes from $i\_0$ to $o\_{\ell}$, which is known as the *forward pass*. And then the weight updating happens from $o\_{\ell}$ to $i\_0$ which is called *backward pass*.
  
  I want to know whether the following mechanism exists in literature assuming that all layers have the same input and output dimensions.
  
  For each iteration
  
  1. Select a subset $L \subseteq \{0, 1, 2, 3, \cdots, \ell\}$ randomly.
  2. Input passes through layers whose indices are present in $L$ only i.e, forward pass happens by dropping some layers.
  3. Update weights for layers whose indices are in $L$ i.e., update the weights of layers which are participated in step (2).
  
  What is the name of the technique mentioned above, if it is present in literature?
  
  **Tags:** neural-networks, training, reference-request, forward-pass
  
  > **Comments:**
  > **username_1**: This should be relatively easy to put together in PyTorch or Chainer (any lib that can dope with dynamic graphs - doing this is in e.g. Tensorflow would be much harder). Have you tried it, does it have any useful impact? I'd imagine it may work as a form of regularisation. It also has *some* relationship to unrolled RNNs, as each layer will tend to work as if it is processing the same kind of input. However, the technique is unlikely to have any references or have been given a name if the results are not good.  
  > **username_2**: The closest technique to what you're proposing that comes to my mind right now is _dropout_, but, in dropout, we _randomly_ drop some neurons from the neural network, so not full layers.  
  > **username_0**: @username_1 I didn't try yet. Need to do.  
  > **username_3**: With some playing around, you could probably find some scenarios where this benefits the learning of a model. Unfortunately, I don't think this method will ever outperform dropout as it simply drops too much at once. No definite answer here though, you'll need to do your own testing to evaluate it's effectiveness  
  
  ---
  
  ## 0 answers
  
  