Answers.Body: <p>We can think of the dropout as of averaging over many networks. Not as a way to mute or denoise your data. Therefore the rule of thumb is the dropout should be applied to higher layers in your network. It is usually applied to fully connected layers (if at all!). It is a common practice to keep 0.5-0.75. neurons active. Personally I prefer L2 much more.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-15T01:22:11.730Z
Answers.Text: So dropout should decrease as we get closer to the output.  Why do you prefer L2?
Answers.UserDisplayName: wordsforthewise
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-15T12:37:33.907Z
Answers.Text: 1. It seems to be more stable in general.
  2. When you have a large pretrained network and you want to keep the architecture unchanged and you want to use it to work with a smaller amount of data - I've empirically found that with little data you can apply strong L2 regularization with v. good effects (to fc layers - not to conv layers). The same doesn't apply to dropout. Keeping less than 0.5 neurons may result in v. unstable training.
Answers.UserDisplayName: FunkyKowal
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-15T20:40:32.933Z
Answers.Text: OK thanks. Do you know if it's beneficial to have dropout on more than one layer, or is one usually good enough?
Answers.UserDisplayName: wordsforthewise
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-15T20:41:04.883Z
Answers.Text: Related question: is it best to use L2 on just 1 layer or all fully_connected?
Answers.UserDisplayName: wordsforthewise
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-16T13:46:20.240Z
Answers.Text: I don't think I've ever thought of applying l2 to only some of fc layers. Although, since you seem like looking for a general advice I would suggest getting the architecture right without focusing on the details too much, and getting to know your architecture through experimenting (on the large changes first, keep it simple).
Answers.UserDisplayName: FunkyKowal
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-14T14:28:26.940Z
Answers.LastActivityDate: 2017-04-14T14:28:26.940Z
Answers.UserDisplayName: FunkyKowal
Body: <p>I know I've seen this somewhere before, but can't find it now.  Say we have a neural network with a handful of layers, and we're applying dropout to each layer.  As we move closer to the output, should dropout decrease, increase, or stay the same?</p>
  
Tags: <neural-networks>
Title: How should dropout change with network depth?
UserDisplayName: wordsforthewise
ThreadText: # How should dropout change with network depth?
  
  (Asked by: username_0 on 2017-03-05)
  
  I know I've seen this somewhere before, but can't find it now. Say we have a neural network with a handful of layers, and we're applying dropout to each layer. As we move closer to the output, should dropout decrease, increase, or stay the same?
  
  **Tags:** neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-04-14. Score: 1 | Accepted answer
  
  We can think of the dropout as of averaging over many networks. Not as a way to mute or denoise your data. Therefore the rule of thumb is the dropout should be applied to higher layers in your network. It is usually applied to fully connected layers (if at all!). It is a common practice to keep 0.5-0.75. neurons active. Personally I prefer L2 much more.
  
  > **Comments:**
  > **username_0**: So dropout should decrease as we get closer to the output.  Why do you prefer L2?  
  > **username_1**: 1. It seems to be more stable in general. 2. When you have a large pretrained network and you want to keep the architecture unchanged and you want to use it to work with a smaller amount of data - I've empirically found that with little data you can apply strong L2 regularization with v. good effects (to fc layers - not to conv layers). The same doesn't apply to dropout. Keeping less than 0.5 neurons may result in v. unstable training.  
  > **username_0**: OK thanks. Do you know if it's beneficial to have dropout on more than one layer, or is one usually good enough?  
  > **username_0**: Related question: is it best to use L2 on just 1 layer or all fully_connected?  
  > **username_1**: I don't think I've ever thought of applying l2 to only some of fc layers. Although, since you seem like looking for a general advice I would suggest getting the architecture right without focusing on the details too much, and getting to know your architecture through experimenting (on the large changes first, keep it simple).  
  
  ---
  
  