Answers.Body: <p>I am also wondering about this.
  It must depend both on convolutional sub-network output size (N) and number of classes (M).
  Maybe there are some rules of thumbs depending on (N, M).</p>
  <ol>
  <li>Why 2 dense layers and not, say, 3 or 4 ?</li>
  <li>Is it better to have all dense layers (except last) the same size ? or decreasing ? or increasing ? or pyramidal ?</li>
  <li>Is it better to have small dense layers or larger ones with dropout between layers ?</li>
  </ol>
  <p>And bonus question:</p>
  <ol start="4">
  <li>Should we use batch normalization between dense layers ?</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-18T12:17:39.980Z
Answers.LastActivityDate: 2021-02-18T12:17:39.980Z
Answers.UserDisplayName: lostdatum
Answers.Body: <p>It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also  2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-20T11:22:44.220Z
Answers.Text: Is it based on your own experience something else ? and why would 2 layers of 512  be better than for example a layer of 4096 ?
Answers.UserDisplayName: Hadrien Berthier
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-20T13:22:05.357Z
Answers.Text: Just common practice. Historically 2 dense layers put on top of VGG/Inception. It works, so everyone use it. Intuition behind 2 layers instead of 1 bigger is that it provide more nonlinearity. But it's not proven. Also the number of weights would be quite different. It's different balance between weights and input/output pipe size
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-20T14:03:34.720Z
Answers.Text: okay thanks :) I was wondering because I saw that a lot but didn't see anything written about
Answers.UserDisplayName: Hadrien Berthier
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-20T11:20:55.723Z
Answers.LastActivityDate: 2019-03-20T11:20:55.723Z
Answers.UserDisplayName: mirror2image
Body: <p>I am fine-tuning a VGG16 model on 20 classes with 500k images I was wondering how do you chose the size of the dense layer (the one before the prediction layer which has a size 20). I would prefer not to do a grid search seeing how long it take to train my model.</p>
  
  <p>Also how many Dense layer should I put after my global average pooling ?</p>
  
  <pre><code>base_model = keras.applications.VGG16(weights='imagenet', include_top=False)
  
    x = base_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(???, activation='relu')(x)
    x = Dropout(0.5, name='drop_fc1')(x)
    prediction_layer = Dense(class_number, activation='softmax')(x)
  </code></pre>
  
  <p>I haven't see particular rules about how its done, are there any ?
  Is it link with the size of the convolution layer ?</p>
  
Tags: <deep-learning><convolutional-neural-networks><hidden-layers>
Title: How to chose dense layer size?
UserDisplayName: Hadrien Berthier
ThreadText: # How to chose dense layer size?
  
  (Asked by: username_0 on 2019-03-20)
  
  I am fine-tuning a VGG16 model on 20 classes with 500k images I was wondering how do you chose the size of the dense layer (the one before the prediction layer which has a size 20). I would prefer not to do a grid search seeing how long it take to train my model.
  
  Also how many Dense layer should I put after my global average pooling ?
  
  ```
  base_model = keras.applications.VGG16(weights='imagenet', include_top=False)
  
    x = base_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(???, activation='relu')(x)
    x = Dropout(0.5, name='drop_fc1')(x)
    prediction_layer = Dense(class_number, activation='softmax')(x)
  
  ```
  
  I haven't see particular rules about how its done, are there any ?
  Is it link with the size of the convolution layer ?
  
  **Tags:** deep-learning, convolutional-neural-networks, hidden-layers
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-03-20. Score: 3 | Accepted answer
  
  It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.
  
  > **Comments:**
  > **username_0**: Is it based on your own experience something else ? and why would 2 layers of 512  be better than for example a layer of 4096 ?  
  > **username_2**: Just common practice. Historically 2 dense layers put on top of VGG/Inception. It works, so everyone use it. Intuition behind 2 layers instead of 1 bigger is that it provide more nonlinearity. But it's not proven. Also the number of weights would be quite different. It's different balance between weights and input/output pipe size  
  > **username_0**: okay thanks :) I was wondering because I saw that a lot but didn't see anything written about  
  
  ---
  
  ### Answer by username_1 on 2021-02-18. Score: 3
  
  I am also wondering about this.
  It must depend both on convolutional sub-network output size (N) and number of classes (M).
  Maybe there are some rules of thumbs depending on (N, M).
  
  1. Why 2 dense layers and not, say, 3 or 4 ?
  2. Is it better to have all dense layers (except last) the same size ? or decreasing ? or increasing ? or pyramidal ?
  3. Is it better to have small dense layers or larger ones with dropout between layers ?
  
  And bonus question:
  
  4. Should we use batch normalization between dense layers ?
  
  
  ---
  
  