Answers.Body: <p>Your issue may be your loss function.</p>
  <blockquote>
  <p>I have been using dropout though to address a regression neural network that I wish to output coordinates (of a 10x10 grid)</p>
  </blockquote>
  <pre><code>---architecture---
  10x10x3                  Input image
  ...
  100x1x1                  Fully connected layer: 100
  2x1x1                    Fully connected layer: 2
  ------------------
  </code></pre>
  <p>It appears like your network is outputting an <span class="math-container">$(x, y)$</span> coordinate in the grid, and you are using the mean-squared error as your loss function. Dropout on the final layer will greatly increase the variance of your output, and the loss. In particular, even if your model at test-time is perfect, your training standard deviation is still at least <span class="math-container">$$\sqrt{\frac{\overbrace{100}^\text{final hidden dimension} \text{dropout}\cdot(1-\text{dropout})}{\text{batch_size}}}=70\%$$</span>
  of the correct value for a dropout of 0.2 and batch size of 32. My recommendations are</p>
  <ol>
  <li>Treat this as a classification problem: softmax <span class="math-container">$10\times 10$</span> outputs, asking which pixel is correct.</li>
  <li>Do not use dropout on the final layer.</li>
  <li>If you can get more data, use a larger batch size.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-27T21:22:35.570Z
Answers.Text: Note that you can add a Gaussian blur to your classifications if you want to keep the mean-squared error prior. I.e., draw a white pixel at the correct location with zeros everywhere else, and blur the image. This rewards your network for getting close, even if not exactly correct.
Answers.UserDisplayName: programjames
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T04:22:44.137Z
Answers.LastActivityDate: 2024-03-26T04:22:44.137Z
Answers.UserDisplayName: programjames
Answers.Body: <pre><code>why dropout of 0.5 effectively kills the networks ability to train.
  </code></pre>
  
  <p>because that is too mutch normal values are like 0.15-0.05. Imagine, the 50% of input image is randomly set to 0, THEN it happens on next layer, means in average 25%  remains, etc... also if you have small dataset with too different images for each class, this + drouput wil confuse the network. 
  Also your CNN setup is not realy rational. Too much fc layers, replace one or two fc to convo layers.  And i d say the reason of using 4x4 is only with stride 2, else use 3x3. And you sould use batch normalisation and augmentation like small noice would be probably petter then dropout in your case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-22T03:05:50.293Z
Answers.Text: From the paper that presented this technique: "In the simplest case, each unit is retained with
  a fixed probability p independent of other units, where p can be chosen using a validation
  set or can simply be set at 0.5, which seems to be close to optimal for a wide range of
  networks and tasks", http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf introduction, paragraph 4
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-06T06:14:55.827Z
Answers.Text: Why are you creating a custom dropout. Why not use the keras dropout layer after each dense layer except of course not the last layer? Please provide actual code you used to create the model.
Answers.UserDisplayName: Gerry P
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-26T08:30:26.630Z
Answers.Text: @Recessive , your JMLR is returning a 404 error. I believe the right one might be this: https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf
Answers.UserDisplayName: evaristegd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-18T10:23:17.247Z
Answers.LastActivityDate: 2019-07-18T10:23:17.247Z
Answers.UserDisplayName: user8426627
Body: <p>I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly having no effect on the error at all:</p>
  
  <p>The following runs were seeded.</p>
  
  <p>key: dropout = 0.3, means 30% chance of dropout</p>
  
  <p>graph x axis: iteration</p>
  
  <p>y axis: error</p>
  
  <p>dropout=0
  <a href="https://i.sstatic.net/bOctt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bOctt.png" alt="enter image description here"></a></p>
  
  <p>dropout = 0.001<a href="https://i.sstatic.net/DKau9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DKau9.png" alt="enter image description here"></a></p>
  
  <p>dropout = 0.1<a href="https://i.sstatic.net/0uUjJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0uUjJ.png" alt="enter image description here"></a></p>
  
  <p>dropout = 0.5
  <a href="https://i.sstatic.net/D3dhv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/D3dhv.png" alt="enter image description here"></a></p>
  
  <p>I don't quite understand why dropout of 0.5 effectively kills the networks ability to train. This specific network here is rather small, a CNN of architecture:</p>
  
  <pre><code>3x3x3                    Input image
  3x3x3                    Convolutional layer: 3x3x3, stride = 1, padding = 1
  20x1x1                   Flatten layer: 27 -&gt; 20
  20x1x1                   Fully connected layer: 20
  10x1x1                   Fully connected layer: 10
  2x1x1                    Fully connected layer: 2
  </code></pre>
  
  <p>But I have tested a CNN with architecture:</p>
  
  <pre><code>10x10x3                  Input image
  9x9x12                   Convolutional layer: 4x4x12, stride = 1, padding = 1
  8x8x12                   Max pooling layer: 2x2, stride = 1
  6x6x24                   Convolutional layer: 3x3x24, stride = 1, padding = 0
  5x5x24                   Max pooling layer: 2x2, stride = 1
  300x1x1                  Flatten layer: 600 -&gt; 300
  300x1x1                  Fully connected layer: 300
  100x1x1                  Fully connected layer: 100
  2x1x1                    Fully connected layer: 2
  </code></pre>
  
  <p>overnight with dropout = 0.2 and it completely failed to learn anything, having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85%. I would just like to know if there's a specific reason as to why this might be happening. My implementation of dropout is as follows:</p>
  
  <p><code>activation = relu(val)*(random.random() &gt; self.dropout)</code></p>
  
  <p>then at test time:</p>
  
  <p><code>activation = relu(val)*(1-self.dropout)</code></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T04:12:22.997Z
Comments.Text: @DuttaA I'm using batch sizes of 32 on a data set of about 1000 images
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T04:14:05.963Z
Comments.Text: Dropout is an effective way to utilize an overparameterized model by averaging multiple models that cover subsets of an overarching feature space.  
  
  In practice its used as a tool against overfitting. In your case you dont even mention test set error. I would reccomend looking more into what dropout is, and what you want from using it.
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T04:41:48.177Z
Comments.Text: @mshlis Sorry, I didn't specify, when I said " having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85% " I was referring to test set error. I took an accurate classification as above 0.65 for the correct output node (with a softmax activation). I have been using dropout though to address a regression neural network that I wish to output coordinates (of a 10x10 grid), but instead seems to just take the average (x, y) coordinates of the training data, which I was guessing was over fitting, hence I am trying dropout.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T04:45:21.650Z
Comments.Text: @mshlis on another note, I took to visualising the output of each convolutional layer by turning the output into a grey scaled image. In doing this, it seems the network is only using roughly 20% of the kernels, as the other 80% are completely black for nearly all inputs. Correct me if I'm wrong, but I believe a good way to combat this would be dropout?
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-26T17:32:40.427Z
Comments.Text: I think you have too few neurons for dropout
Comments.UserDisplayName: CoffeDeveloper
Tags: <neural-networks><convolutional-neural-networks><dropout><regularization><relu>
Title: Dropout causes too much noise for network to train
UserDisplayName: Recessive
ThreadText: # Dropout causes too much noise for network to train
  
  (Asked by: username_0 on 2019-07-16)
  
  I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly having no effect on the error at all:
  The following runs were seeded.
  key: dropout = 0.3, means 30% chance of dropout
  graph x axis: iteration
  y axis: error
  dropout=0
  
  dropout = 0.001
  dropout = 0.1
  dropout = 0.5
  
  I don't quite understand why dropout of 0.5 effectively kills the networks ability to train. This specific network here is rather small, a CNN of architecture:
  3x3x3                    Input image
  3x3x3                    Convolutional layer: 3x3x3, stride = 1, padding = 1
  20x1x1                   Flatten layer: 27 -> 20
  20x1x1                   Fully connected layer: 20
  10x1x1                   Fully connected layer: 10
  2x1x1                    Fully connected layer: 2
  
  But I have tested a CNN with architecture:
  10x10x3                  Input image
  9x9x12                   Convolutional layer: 4x4x12, stride = 1, padding = 1
  8x8x12                   Max pooling layer: 2x2, stride = 1
  6x6x24                   Convolutional layer: 3x3x24, stride = 1, padding = 0
  5x5x24                   Max pooling layer: 2x2, stride = 1
  300x1x1                  Flatten layer: 600 -> 300
  300x1x1                  Fully connected layer: 300
  100x1x1                  Fully connected layer: 100
  2x1x1                    Fully connected layer: 2
  
  overnight with dropout = 0.2 and it completely failed to learn anything, having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85%. I would just like to know if there's a specific reason as to why this might be happening. My implementation of dropout is as follows:
  activation = relu(val)*(random.random() > self.dropout)
  then at test time:
  activation = relu(val)*(1-self.dropout)
  
  **Tags:** neural-networks, convolutional-neural-networks, dropout, regularization, relu
  > **Comments:**
  > **username_0**: @DuttaA I'm using batch sizes of 32 on a data set of about 1000 images  
  > **username_1**: Dropout is an effective way to utilize an overparameterized model by averaging multiple models that cover subsets of an overarching feature space.    In practice its used as a tool against overfitting. In your case you dont even mention test set error. I would reccomend looking more into what dropout is, and what you want from using it.  
  > **username_0**: @mshlis Sorry, I didn't specify, when I said " having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85% " I was referring to test set error. I took an accurate classification as above 0.65 for the correct output node (with a softmax activation). I have been using dropout though to address a regression neural network that I wish to output coordinates (of a 10x10 grid), but instead seems to just take the average (x, y) coordinates of the training data, which I was guessing was over fitting, hence I am trying dropout.  
  > **username_0**: @mshlis on another note, I took to visualising the output of each convolutional layer by turning the output into a grey scaled image. In doing this, it seems the network is only using roughly 20% of the kernels, as the other 80% are completely black for nearly all inputs. Correct me if I'm wrong, but I believe a good way to combat this would be dropout?  
  > **username_2**: I think you have too few neurons for dropout  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-03-26. Score: 0
  
  Your issue may be your loss function.
  
  I have been using dropout though to address a regression neural network that I wish to output coordinates (of a 10x10 grid)
  
  ---architecture---
  10x10x3                  Input image
  ...
  100x1x1                  Fully connected layer: 100
  2x1x1                    Fully connected layer: 2
  ------------------
  
  It appears like your network is outputting an $(x, y)$ coordinate in the grid, and you are using the mean-squared error as your loss function. Dropout on the final layer will greatly increase the variance of your output, and the loss. In particular, even if your model at test-time is perfect, your training standard deviation is still at least $$\sqrt{\frac{\overbrace{100}^\text{final hidden dimension} \text{dropout}\cdot(1-\text{dropout})}{\text{batch_size}}}=70\%$$
  of the correct value for a dropout of 0.2 and batch size of 32. My recommendations are
  
  Treat this as a classification problem: softmax $10\times 10$ outputs, asking which pixel is correct.
  Do not use dropout on the final layer.
  If you can get more data, use a larger batch size.
  
  > **Comments:**
  > **username_3**: Note that you can add a Gaussian blur to your classifications if you want to keep the mean-squared error prior. I.e., draw a white pixel at the correct location with zeros everywhere else, and blur the image. This rewards your network for getting close, even if not exactly correct.  
  
  ---
  
  ### Answer by username_4 on 2019-07-18. Score: 0
  
  why dropout of 0.5 effectively kills the networks ability to train.
  
  because that is too mutch normal values are like 0.15-0.05. Imagine, the 50% of input image is randomly set to 0, THEN it happens on next layer, means in average 25%  remains, etc... also if you have small dataset with too different images for each class, this + drouput wil confuse the network. 
  Also your CNN setup is not realy rational. Too much fc layers, replace one or two fc to convo layers.  And i d say the reason of using 4x4 is only with stride 2, else use 3x3. And you sould use batch normalisation and augmentation like small noice would be probably petter then dropout in your case.
  
  > **Comments:**
  > **username_0**: From the paper that presented this technique: "In the simplest case, each unit is retained with a fixed probability p independent of other units, where p can be chosen using a validation set or can simply be set at 0.5, which seems to be close to optimal for a wide range of networks and tasks", http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf introduction, paragraph 4  
  > **username_5**: Why are you creating a custom dropout. Why not use the keras dropout layer after each dense layer except of course not the last layer? Please provide actual code you used to create the model.  
  > **username_6**: @Recessive , your JMLR is returning a 404 error. I believe the right one might be this: https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf  
  
  ---
  
  