<p>I am also wondering about this.
It must depend both on convolutional sub-network output size (N) and number of classes (M).
Maybe there are some rules of thumbs depending on (N, M).</p>
<ol>
<li>Why 2 dense layers and not, say, 3 or 4 ?</li>
<li>Is it better to have all dense layers (except last) the same size ? or decreasing ? or increasing ? or pyramidal ?</li>
<li>Is it better to have small dense layers or larger ones with dropout between layers ?</li>
</ol>
<p>And bonus question:</p>
<ol start="4">
<li>Should we use batch normalization between dense layers ?</li>
</ol>

CC BY-SA 4.0
2021-02-18T12:17:39.980Z
2021-02-18T12:17:39.980Z
lostdatum
<p>It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also  2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.</p>

CC BY-SA 4.0
2019-03-20T11:22:44.220Z
Is it based on your own experience something else ? and why would 2 layers of 512  be better than for example a layer of 4096 ?
Hadrien Berthier
CC BY-SA 4.0
2019-03-20T13:22:05.357Z
Just common practice. Historically 2 dense layers put on top of VGG/Inception. It works, so everyone use it. Intuition behind 2 layers instead of 1 bigger is that it provide more nonlinearity. But it's not proven. Also the number of weights would be quite different. It's different balance between weights and input/output pipe size
mirror2image
CC BY-SA 4.0
2019-03-20T14:03:34.720Z
okay thanks :) I was wondering because I saw that a lot but didn't see anything written about
Hadrien Berthier
CC BY-SA 4.0
2019-03-20T11:20:55.723Z
2019-03-20T11:20:55.723Z
mirror2image
<p>I am fine-tuning a VGG16 model on 20 classes with 500k images I was wondering how do you chose the size of the dense layer (the one before the prediction layer which has a size 20). I would prefer not to do a grid search seeing how long it take to train my model.</p>

<p>Also how many Dense layer should I put after my global average pooling ?</p>

<pre><code>base_model = keras.applications.VGG16(weights='imagenet', include_top=False)

  x = base_model.output
  x = GlobalAveragePooling2D()(x)
  x = Dense(???, activation='relu')(x)
  x = Dropout(0.5, name='drop_fc1')(x)
  prediction_layer = Dense(class_number, activation='softmax')(x)
</code></pre>

<p>I haven't see particular rules about how its done, are there any ?
Is it link with the size of the convolution layer ?</p>

<deep-learning><convolutional-neural-networks><hidden-layers>
How to chose dense layer size?
Hadrien Berthier
# How to chose dense layer size?

(Asked by: username_0 on 2019-03-20)

I am fine-tuning a VGG16 model on 20 classes with 500k images I was wondering how do you chose the size of the dense layer (the one before the prediction layer which has a size 20). I would prefer not to do a grid search seeing how long it take to train my model.
Also how many Dense layer should I put after my global average pooling ?
base_model = keras.applications.VGG16(weights='imagenet', include_top=False)

  x = base_model.output
  x = GlobalAveragePooling2D()(x)
  x = Dense(???, activation='relu')(x)
  x = Dropout(0.5, name='drop_fc1')(x)
  prediction_layer = Dense(class_number, activation='softmax')(x)

I haven't see particular rules about how its done, are there any ?
Is it link with the size of the convolution layer ?

**Tags:** deep-learning, convolutional-neural-networks, hidden-layers


---

## 2 answers

### Answer by username_2 on 2019-03-20. Score: 3 | Accepted answer

It's depend more on number of classes. For 20 classes 2 layers 512 should be more then enough. If you want to experiment you can try also 2 x 256 and 2 x 1024. Less then 256 may work too, but you may underutilize power of previous conv layers.

> **Comments:**
> **username_0**: Is it based on your own experience something else ? and why would 2 layers of 512  be better than for example a layer of 4096 ?  
> **username_2**: Just common practice. Historically 2 dense layers put on top of VGG/Inception. It works, so everyone use it. Intuition behind 2 layers instead of 1 bigger is that it provide more nonlinearity. But it's not proven. Also the number of weights would be quite different. It's different balance between weights and input/output pipe size  
> **username_0**: okay thanks :) I was wondering because I saw that a lot but didn't see anything written about  

---

### Answer by username_1 on 2021-02-18. Score: 3

I am also wondering about this.
It must depend both on convolutional sub-network output size (N) and number of classes (M).
Maybe there are some rules of thumbs depending on (N, M).

1. Why 2 dense layers and not, say, 3 or 4 ?
2. Is it better to have all dense layers (except last) the same size ? or decreasing ? or increasing ? or pyramidal ?
3. Is it better to have small dense layers or larger ones with dropout between layers ?

And bonus question:

4. Should we use batch normalization between dense layers ?


---

