Answers.Body: <p>I think we would consider regularization and downsampling better in this way:</p>
  
  <ol>
  <li>dropout</li>
  </ol>
  
  <p>it puts some input value (neuron) for the next layer as 0, which makes the current layer a sparse one. So it reduces the dependence of each feature in this layer.</p>
  
  <ol>
  <li>pooling layer</li>
  </ol>
  
  <p>the downsampling directly remove some input, and that makes the layer "smaller" rather than "sparser". The difference can be subtle but clear enough.</p>
  
  <p>That's the root reason why the former also affect the evaluation/test process but the later does not. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-12T16:26:41.420Z
Answers.LastActivityDate: 2019-02-12T16:26:41.420Z
Answers.UserDisplayName: Bs He
Answers.Body: <p>Dropout and Max-pooling are performed for different reasons.</p>
  
  <p>Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple terms, it helps the model to learn that some features are an "OR" and not an "AND" requirements.</p>
  
  <p>Max-pooling is not a regularization technique and it is part of the model's architecture, so it is also used during evaluation. The goal of max-pooling is to down-sample an input representation. As a result the model becomes less sensitive to some translations (improving translation invariance).</p>
  
  <p>As for your last question, yes. dropout can be used after a pooling layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-17T13:54:10.090Z
Answers.Text: Clean answer thank you. If I understood well, features maps in CNN are linked through convolution and pooling layers. Where could we introduce the drop out ? (In dense connection weights are obvious, and so is  the gain in sparsity)
Answers.UserDisplayName: nsaura
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-18T09:20:46.797Z
Answers.Text: The dropout would be used after the pooling layers. I will say that it is very uncommon to use dropout in the convolution layers. In the convolution layers it can be viewed as adding occlusion, which helps the model to focus on smaller features.
Answers.UserDisplayName: Mark.F
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-17T12:43:42.323Z
Answers.LastActivityDate: 2018-12-17T12:43:42.323Z
Answers.UserDisplayName: Mark.F
Body: <p>If I got well the idea of dropout, it allows improving the sparsity of the information that comes from one layer to another by setting some weights to zero.</p>
  <p>On the other hand, pooling, let's say max-pooling, takes the maximum value in a neighborhood, reducing as well to zero, the influence of values apart from this maximum.</p>
  <p>Without considering the shape transformation due to the pooling layer, can we say that pooling is a kind of a dropout step?</p>
  <p>Would the addition of a dropout (or DropConnect) layer, after a pooling layer, make sense in a CNN? And does it help the training process and generalization property?</p>
  
Tags: <convolutional-neural-networks><training><dropout><generalization><pooling>
Title: Is pooling a kind of dropout?
UserDisplayName: nsaura
ThreadText: # Is pooling a kind of dropout?
  
  (Asked by: username_0 on 2018-12-17)
  
  If I got well the idea of dropout, it allows improving the sparsity of the information that comes from one layer to another by setting some weights to zero.
  
  On the other hand, pooling, let's say max-pooling, takes the maximum value in a neighborhood, reducing as well to zero, the influence of values apart from this maximum.
  
  Without considering the shape transformation due to the pooling layer, can we say that pooling is a kind of a dropout step?
  
  Would the addition of a dropout (or DropConnect) layer, after a pooling layer, make sense in a CNN? And does it help the training process and generalization property?
  
  **Tags:** convolutional-neural-networks, training, dropout, generalization, pooling
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-12-17. Score: 8 | Accepted answer
  
  Dropout and Max-pooling are performed for different reasons.
  
  Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple terms, it helps the model to learn that some features are an "OR" and not an "AND" requirements.
  
  Max-pooling is not a regularization technique and it is part of the model's architecture, so it is also used during evaluation. The goal of max-pooling is to down-sample an input representation. As a result the model becomes less sensitive to some translations (improving translation invariance).
  
  As for your last question, yes. dropout can be used after a pooling layer.
  
  > **Comments:**
  > **username_0**: Clean answer thank you. If I understood well, features maps in CNN are linked through convolution and pooling layers. Where could we introduce the drop out ? (In dense connection weights are obvious, and so is  the gain in sparsity)  
  > **username_2**: The dropout would be used after the pooling layers. I will say that it is very uncommon to use dropout in the convolution layers. In the convolution layers it can be viewed as adding occlusion, which helps the model to focus on smaller features.  
  
  ---
  
  ### Answer by username_1 on 2019-02-12. Score: 2
  
  I think we would consider regularization and downsampling better in this way:
  
  1. dropout
  
  it puts some input value (neuron) for the next layer as 0, which makes the current layer a sparse one. So it reduces the dependence of each feature in this layer.
  
  1. pooling layer
  
  the downsampling directly remove some input, and that makes the layer "smaller" rather than "sparser". The difference can be subtle but clear enough.
  
  That's the root reason why the former also affect the evaluation/test process but the later does not.
  
  
  ---
  
  