<p>It depends. It could give you a boost or it could not.</p>

<p>Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches). </p>

<p><strong>Ideal World:</strong> We optimize the network as a whole to gain better course grained features over the sequential layers of the encoder</p>

<p><strong>Reality years ago:</strong> Deep nets have trouble in information propagation either forward or backwards (ex: vanishing/exploding activation and vanishing/exploding gradient).  </p>

<ul>
<li><strong>solution:</strong> Break up the training into iterative schemes that don't require backward information to propagate far at each optimization step

<ul>
<li><strong>Cons:</strong> Each step is looking for a greedy solution, rather than a deep one.</li>
</ul></li>
</ul>

<p><strong>Reality today:</strong> Depending on the domain, there are various publications discussing how to circumvent this issue (ex: Residual networks, He initialization, fixup initialization, batchnorms, different activation functions, etc...)</p>

<ul>
<li><strong>solution:</strong> We can train deep AEs without sequential layerwise training and achieve usually better results, because the model and optimization scheme allow for deep representations to form.</li>
</ul>

<p>I hope this  helped give some form of intuition of the matter.</p>

CC BY-SA 4.0
2020-01-17T15:57:39.543Z
2020-01-17T15:57:39.543Z
mshlis
<p>I was looking into the use of a greedy layer-wise pretraining to initialize the weights of my network.</p>

<p>Just for the sake of clarity: I'm referring to the use of gradually deeper and deeper autoencoders to teach the network gradually more abstract representations of the input one layer at the time.</p>

<p>However, reading <a href="https://machinelearningmastery.com/greedy-layer-wise-pretraining-tutorial/" rel="nofollow noreferrer"><strong>HERE</strong></a>, I read:</p>

<blockquote>
  <p>Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.</p>
</blockquote>

<p>and</p>

<blockquote>
  <p>Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.</p>
</blockquote>

<p>My question is then: if I'm building a network already using "<em>modern</em>" techniques, such as ReLU activations, batch normalization, adam optimizers, etc, is the good-ol' greedy layer-wise pretraining useless? Or can it still provide an edge in the initialization of the network?</p>

<neural-networks>
Is greedy layer-wise pretraining obsolete?
Luca
# Is greedy layer-wise pretraining obsolete?

(Asked by: username_0 on 2020-01-16)

I was looking into the use of a greedy layer-wise pretraining to initialize the weights of my network.

Just for the sake of clarity: I'm referring to the use of gradually deeper and deeper autoencoders to teach the network gradually more abstract representations of the input one layer at the time.

However, reading [**HERE**](https://machinelearningmastery.com/greedy-layer-wise-pretraining-tutorial/), I read:

> Nevertheless, it is likely better performance may be achieved using modern methods such as better activation functions, weight initialization, variants of gradient descent, and regularization methods.

and

> Today, we now know that greedy layer-wise pretraining is not required to train fully connected deep architectures, but the unsupervised pretraining approach was the first method to succeed.

My question is then: if I'm building a network already using "*modern*" techniques, such as ReLU activations, batch normalization, adam optimizers, etc, is the good-ol' greedy layer-wise pretraining useless? Or can it still provide an edge in the initialization of the network?

**Tags:** neural-networks


---

## 1 answers

### Answer by username_1 on 2020-01-17. Score: 2 | Accepted answer

It depends. It could give you a boost or it could not.

Intuitively I would expect it to actually hurt performance if the network is initialized correctly (I think the optimizer is less of a bottleneck because they will have the same effect in both approaches).

**Ideal World:** We optimize the network as a whole to gain better course grained features over the sequential layers of the encoder

**Reality years ago:** Deep nets have trouble in information propagation either forward or backwards (ex: vanishing/exploding activation and vanishing/exploding gradient).

* **solution:** Break up the training into iterative schemes that don't require backward information to propagate far at each optimization step
  + **Cons:** Each step is looking for a greedy solution, rather than a deep one.

**Reality today:** Depending on the domain, there are various publications discussing how to circumvent this issue (ex: Residual networks, He initialization, fixup initialization, batchnorms, different activation functions, etc...)

* **solution:** We can train deep AEs without sequential layerwise training and achieve usually better results, because the model and optimization scheme allow for deep representations to form.

I hope this helped give some form of intuition of the matter.


---

