Body: <p>I just had a very strange experience where I was training an 8 layer deep &amp; pretty wide (max: 512 neurons) neural network for a regression task. I had assumed since it was big enough that it would benefit from some combination of dropout, batch norm, and/or skip layers. And I expected all 3 would be best. I turns out however that each one of them <em>made the model perform much worse</em>, together or nearly in any combination (even at 10% dropout rate).</p>
  <p>This made me wonder when you can expect these methods to help the model vs hurt? I thought they were pretty universal before this (if the model is big enough)... It might be related to this data's strange distribution, I think it is exponential (except it can dip negative sometimes). But scaled to stay within 0-1.</p>
  <p>P.S. Similar experience with log transforming the data!</p>
  <p>I've attach a histogram of the predicted variable for reference in my example (log scale using only positive values, which are the majority).
  <a href="https://i.sstatic.net/KWAHE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KWAHE.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-07T09:40:12.613Z
Comments.Text: From prior experience, there really isn't a robust method of knowing when such methods will be effective or not. In my opinion, methods like dropout or batch norm aren't "standard tricks" like many believe. They should be applied based on strictly experimental results rather than vague theories behind them. 
  
  Even the theory behind them aren't exactly robust. For example, the reason behind batch norm's success is still controversial (https://arxiv.org/abs/2102.06171).
Comments.UserDisplayName: DKDK
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-07T09:40:15.600Z
Comments.Text: Also, I have seen many cases in regression tasks where batch norm and dropout are hurtful. See (https://towardsdatascience.com/pitfalls-with-dropout-and-batchnorm-in-regression-problems-39e02ce08e4d) for more detail.
Comments.UserDisplayName: DKDK
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-09T19:56:55.360Z
Comments.Text: These are hyper-parameters of the model, and you need to use popular methods like Bayesian Optimization to find the best value for them.
Comments.UserDisplayName: OmG
Tags: <regularization>
Title: What determines when Dropout, BatchNorm & other Regularization will be effective?
UserDisplayName: profPlum
ThreadText: # What determines when Dropout, BatchNorm & other Regularization will be effective?
  
  (Asked by: username_0 on 2022-01-07)
  
  I just had a very strange experience where I was training an 8 layer deep & pretty wide (max: 512 neurons) neural network for a regression task. I had assumed since it was big enough that it would benefit from some combination of dropout, batch norm, and/or skip layers. And I expected all 3 would be best. I turns out however that each one of them *made the model perform much worse*, together or nearly in any combination (even at 10% dropout rate).
  
  This made me wonder when you can expect these methods to help the model vs hurt? I thought they were pretty universal before this (if the model is big enough)... It might be related to this data's strange distribution, I think it is exponential (except it can dip negative sometimes). But scaled to stay within 0-1.
  
  P.S. Similar experience with log transforming the data!
  
  I've attach a histogram of the predicted variable for reference in my example (log scale using only positive values, which are the majority).
  [![enter image description here](https://i.sstatic.net/KWAHE.png)](https://i.sstatic.net/KWAHE.png)
  
  **Tags:** regularization
  > **Comments:**
  > **username_1**: From prior experience, there really isn't a robust method of knowing when such methods will be effective or not. In my opinion, methods like dropout or batch norm aren't "standard tricks" like many believe. They should be applied based on strictly experimental results rather than vague theories behind them.   Even the theory behind them aren't exactly robust. For example, the reason behind batch norm's success is still controversial (https://arxiv.org/abs/2102.06171).  
  > **username_1**: Also, I have seen many cases in regression tasks where batch norm and dropout are hurtful. See (https://towardsdatascience.com/pitfalls-with-dropout-and-batchnorm-in-regression-problems-39e02ce08e4d) for more detail.  
  > **username_2**: These are hyper-parameters of the model, and you need to use popular methods like Bayesian Optimization to find the best value for them.  
  
  ---
  
  ## 0 answers
  
  