Answers.Body: <p>It depends. It could give you a boost or it could not.</p>
  
  <p>Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches). </p>
  
  <p><strong>Ideal World:</strong> We optimize the network as a whole to gain better course grained features over the sequential layers of the encoder</p>
  
  <p><strong>Reality years ago:</strong> Deep nets have trouble in information propagation either forward or backwards (ex: vanishing/exploding activation and vanishing/exploding gradient).  </p>
  
  <ul>
  <li><strong>solution:</strong> Break up the training into iterative schemes that don't require backward information to propagate far at each optimization step
  
  <ul>
  <li><strong>Cons:</strong> Each step is looking for a greedy solution, rather than a deep one.</li>
  </ul></li>
  </ul>
  
  <p><strong>Reality today:</strong> Depending on the domain, there are various publications discussing how to circumvent this issue (ex: Residual networks, He initialization, fixup initialization, batchnorms, different activation functions, etc...)</p>
  
  <ul>
  <li><strong>solution:</strong> We can train deep AEs without sequential layerwise training and achieve usually better results, because the model and optimization scheme allow for deep representations to form.</li>
  </ul>
  
  <p>I hope this  helped give some form of intuition of the matter.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-17T15:57:39.543Z
Answers.LastActivityDate: 2020-01-17T15:57:39.543Z
Answers.UserDisplayName: mshlis
Body: <p>I was looking into the use of a greedy layer-wise pretraining to initialize the weights of my network.</p>
  
  <p>Just for the sake of clarity: I'm referring to the use of gradually deeper and deeper autoencoders to teach the network gradually more abstract representations of the input one layer at the time.</p>
  
  <p>However, reading <a href="https://machinelearningmastery.com/greedy-layer-wise-pretraining-tutorial/" rel="nofollow noreferrer"><strong>HERE</strong></a>, I read:</p>
  
  <blockquote>
    <p>Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.</p>
  </blockquote>
  
  <p>and</p>
  
  <blockquote>
    <p>Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.</p>
  </blockquote>
  
  <p>My question is then: if I'm building a network already using "<em>modern</em>" techniques, such as ReLU activations, batch normalization, adam optimizers, etc, is the good-ol' greedy layer-wise pretraining useless? Or can it still provide an edge in the initialization of the network?</p>
  
Tags: <neural-networks>
Title: Is greedy layer-wise pretraining obsolete?
UserDisplayName: Luca
ThreadText: # Is greedy layer-wise pretraining obsolete?
  
  (Asked by: username_0 on 2020-01-16)
  
  I was looking into the use of a greedy layer-wise pretraining to initialize the weights of my network.
  
  Just for the sake of clarity: I'm referring to the use of gradually deeper and deeper autoencoders to teach the network gradually more abstract representations of the input one layer at the time.
  
  However, reading [**HERE**](https://machinelearningmastery.com/greedy-layer-wise-pretraining-tutorial/), I read:
  
  > Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.
  
  and
  
  > Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.
  
  My question is then: if I'm building a network already using "*modern*" techniques, such as ReLU activations, batch normalization, adam optimizers, etc, is the good-ol' greedy layer-wise pretraining useless? Or can it still provide an edge in the initialization of the network?
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-17. Score: 2 | Accepted answer
  
  It depends. It could give you a boost or it could not.
  
  Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches).
  
  **Ideal World:** We optimize the network as a whole to gain better course grained features over the sequential layers of the encoder
  
  **Reality years ago:** Deep nets have trouble in information propagation either forward or backwards (ex: vanishing/exploding activation and vanishing/exploding gradient).
  
  * **solution:** Break up the training into iterative schemes that don't require backward information to propagate far at each optimization step
    + **Cons:** Each step is looking for a greedy solution, rather than a deep one.
  
  **Reality today:** Depending on the domain, there are various publications discussing how to circumvent this issue (ex: Residual networks, He initialization, fixup initialization, batchnorms, different activation functions, etc...)
  
  * **solution:** We can train deep AEs without sequential layerwise training and achieve usually better results, because the model and optimization scheme allow for deep representations to form.
  
  I hope this helped give some form of intuition of the matter.
  
  
  ---
  
  