Answers.Body: <p>I suspect there are two principles at play here, but I don't know which principle is more important in this case.</p>
  <p>The first principle is the conventional wisdom: The lower layers not only learn to summarize the visual features found in the training set, but also learn those features which allow discrimination between categories.  For example, in a beach image, there are likely many subtle yellow vs yellow or blue vs blue features that are technically part of the beach image.  But where we sea the line in the sky may be a stronger indication of what this picture is of.  All the features would be important for reconstructing an image of the beach, but the horizon line may be more important for distinguishing the beach from a picture of the sun in the sky for example.  In a CNN like AlexNet, the horizon line would receive more weight early on in training.</p>
  <p>But I don't see how this first principle would apply in the case of an autoencoding network like <a href="https://icml.cc/2012/papers/73.pdf" rel="nofollow noreferrer">Le et al. 2012</a></p>
  <p>The second principle is that random weights actually have some value from the very beginning of training, and those features which are useful early in training for a particular high-level decision will become even more useful later in training as they pick up patterns from the images to which they relate.</p>
  <p>But I again don't see how this applies to an autoencoder like Le et al.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T21:03:50.300Z
Answers.LastActivityDate: 2021-12-27T21:03:50.300Z
Answers.UserDisplayName: Josiah Yoder
Body: <p>This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as <a href="https://icml.cc/2012/papers/73.pdf" rel="nofollow noreferrer">Le et al. 2012</a>.</p>
  <p>What is the advantage of training all the layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt? And why would there ever be an advantage for an autoencoder like <a href="https://icml.cc/2012/papers/73.pdf" rel="nofollow noreferrer">Le et al. 2012</a> where there is no backpropagation to communicate from the later layers to the earlier layers?</p>
  <p>I think the conventional answer is that the lower layers can actually learn to provide low-level features that support the layers above. An example of this is learning to detect a horizontal yellow-blue feature to detect the water line in a beach scene.</p>
  <p>But couldn't the yellow-blue feature be found just as easily by training the lower layers first? This would be especially true of an autoencoder such as <a href="https://icml.cc/2012/papers/73.pdf" rel="nofollow noreferrer">Le et al. 2012</a>, which picks up on patterns in the training set without having ground truth-labels to group them.</p>
  <p>Citations to experiments or theoretical work that directly answers this question would be appreciated!</p>
  <p>This is a follow-on to <a href="https://ai.stackexchange.com/questions/6298/do-le-et-al-2012-train-all-three-autoencoder-layers-at-a-time-or-just-one">an earlier question</a>.</p>
  
Tags: <convolutional-neural-networks><autoencoders><multilayer-perceptrons>
Title: Why is training all layers at a time effective for a multi-layer autoencoder?
UserDisplayName: Josiah Yoder
ThreadText: # Why is training all layers at a time effective for a multi-layer autoencoder?
  
  (Asked by: username_0 on 2021-12-27)
  
  This training of all layers of a CNN simultaneously is standard practice today. It is found in every CNN (AlexNet (2012), VGG, Inception, GANs, etc) and even pre-CNN networks such as [Le et al. 2012](https://icml.cc/2012/papers/73.pdf).
  
  What is the advantage of training all the layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt? And why would there ever be an advantage for an autoencoder like [Le et al. 2012](https://icml.cc/2012/papers/73.pdf) where there is no backpropagation to communicate from the later layers to the earlier layers?
  
  I think the conventional answer is that the lower layers can actually learn to provide low-level features that support the layers above. An example of this is learning to detect a horizontal yellow-blue feature to detect the water line in a beach scene.
  
  But couldn't the yellow-blue feature be found just as easily by training the lower layers first? This would be especially true of an autoencoder such as [Le et al. 2012](https://icml.cc/2012/papers/73.pdf), which picks up on patterns in the training set without having ground truth-labels to group them.
  
  Citations to experiments or theoretical work that directly answers this question would be appreciated!
  
  This is a follow-on to [an earlier question](https://ai.stackexchange.com/questions/6298/do-le-et-al-2012-train-all-three-autoencoder-layers-at-a-time-or-just-one).
  
  **Tags:** convolutional-neural-networks, autoencoders, multilayer-perceptrons
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-12-27. Score: 1
  
  I suspect there are two principles at play here, but I don't know which principle is more important in this case.
  
  The first principle is the conventional wisdom: The lower layers not only learn to summarize the visual features found in the training set, but also learn those features which allow discrimination between categories. For example, in a beach image, there are likely many subtle yellow vs yellow or blue vs blue features that are technically part of the beach image. But where we sea the line in the sky may be a stronger indication of what this picture is of. All the features would be important for reconstructing an image of the beach, but the horizon line may be more important for distinguishing the beach from a picture of the sun in the sky for example. In a CNN like AlexNet, the horizon line would receive more weight early on in training.
  
  But I don't see how this first principle would apply in the case of an autoencoding network like [Le et al. 2012](https://icml.cc/2012/papers/73.pdf)
  
  The second principle is that random weights actually have some value from the very beginning of training, and those features which are useful early in training for a particular high-level decision will become even more useful later in training as they pick up patterns from the images to which they relate.
  
  But I again don't see how this applies to an autoencoder like Le et al.
  
  
  ---
  
  