Answers.Body: <p>The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.</p>
  <p>This also agrees with the fact that only the autoencoder layer has tunable parameters. The other two layers use uniform weights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-07T16:53:52.863Z
Answers.LastActivityDate: 2021-12-24T08:05:33.880Z
Answers.LastEditDate: 2021-12-24T08:05:33.880Z
Answers.UserDisplayName: Josiah Yoder
Body: <p><a href="https://icml.cc/2012/papers/73.pdf" rel="nofollow noreferrer">Le et al. 2012</a> use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).</p>
  <p>Their network is built with three autoregressive layers, and six pooling and normalization layers.</p>
  <p>In the paper they state,</p>
  <blockquote>
  <p><strong>Optimization</strong>: All parameters in our model were
  trained jointly with the objective being the sum of the
  objectives of the three layers.</p>
  </blockquote>
  <p>Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?</p>
  <p>I asked a <a href="https://ai.stackexchange.com/questions/33935/why-is-training-all-layers-at-a-time-effective-for-a-multi-layer-autoencoder">follow-on question</a> on the advantage of training all layer simultaneously.</p>
  
Tags: <papers><autoencoders><unsupervised-learning>
Title: Do Le et al. (2012) train all three autoencoder layers at a time, or just one?
UserDisplayName: Josiah Yoder
ThreadText: # Do Le et al. (2012) train all three autoencoder layers at a time, or just one?
  
  (Asked by: username_0 on 2018-05-07)
  
  [Le et al. 2012](https://icml.cc/2012/papers/73.pdf) use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).
  
  Their network is built with three autoregressive layers, and six pooling and normalization layers.
  
  In the paper they state,
  
  > **Optimization**: All parameters in our model were
  > trained jointly with the objective being the sum of the
  > objectives of the three layers.
  
  Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?
  
  I asked a [follow-on question](https://ai.stackexchange.com/questions/33935/why-is-training-all-layers-at-a-time-effective-for-a-multi-layer-autoencoder) on the advantage of training all layer simultaneously.
  
  **Tags:** papers, autoencoders, unsupervised-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2018-05-07. Score: 2 | Accepted answer
  
  The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.
  
  This also agrees with the fact that only the autoencoder layer has tunable parameters. The other two layers use uniform weights.
  
  
  ---
  
  