Body: <p>There are a few ways to regularise a neural network, for example dropout or the L1. Now, both these methods, and possibly most other regularisation methods, tend to remove from, or simplify the neural network. The Dropout deactivates nodes and the L1 shrinks the weights of the model, and so on.</p>
  <p>The main argument in favour of regularising a neural network is that by simplifying the model you are forcing it to learn more general functions and thus making the neural network more robust to overfitting or noisy input.</p>
  <p>Once you have a model trained with, and without, regularisation, it is possible to compare their performance by calculating the error metrics on their outputs. This will prove whether the regularised model performs better than the standard model or not.</p>
  <p>However, considering that the regularised model achieved better performance on its error metrics, how to prove that the weights of the regularised model have less variance (simpler) than the standard neural network?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-07T06:28:58.200Z
Comments.Text: This question doesn't have an answer unless you define what you mean by complexity. If by complexity you mean the number of parameters, then both regularization methods you mentioned do nothing for complexity. If by complexity you mean the variance in parameters you might have more luck, but I imagine that dropout wouldn't show any reduced complexity.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-07T06:39:14.800Z
Comments.Text: @Recessive Thank you I updated my question. But how does dropout not reduce complexity if it deactivates nodes during the training process? Having less nodes seems like a simpler model.
Comments.UserDisplayName: Marcus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-09T10:02:50.107Z
Comments.Text: It inconsistently deactivates nodes during training which means you need just as many parameters, and then at testing time it doesn't deactivate them at all
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-11T09:45:38.347Z
Comments.Text: Hello. I still think that this question is not clear enough, because of the part. "how to prove that the weights of the regularised model have less variance (simpler) than the standard neural network?". It seems that you're using "simpler" as a synonym for having less variance, but what do you mean by "less variance" here? You need to clarify this part, because, once we have 2 neural networks, we have 2 specific sets of weights, so the only thing that you can compute is really the variance of the values of these parameters.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-11T09:46:14.450Z
Comments.Text: So, are you looking for a theoretical result that shows that, if using e.g. L2 regularization, your parameters will be limited to a smaller set of values, so will have less variance generally?
Comments.UserDisplayName: nbro
Tags: <neural-networks><proofs><regularization><dropout><l1-regularization>
Title: How to prove that a regularisation method simplified a neural network?
UserDisplayName: Marcus
ThreadText: # How to prove that a regularisation method simplified a neural network?
  
  (Asked by: username_0 on 2021-05-07)
  
  There are a few ways to regularise a neural network, for example dropout or the L1. Now, both these methods, and possibly most other regularisation methods, tend to remove from, or simplify the neural network. The Dropout deactivates nodes and the L1 shrinks the weights of the model, and so on.
  
  The main argument in favour of regularising a neural network is that by simplifying the model you are forcing it to learn more general functions and thus making the neural network more robust to overfitting or noisy input.
  
  Once you have a model trained with, and without, regularisation, it is possible to compare their performance by calculating the error metrics on their outputs. This will prove whether the regularised model performs better than the standard model or not.
  
  However, considering that the regularised model achieved better performance on its error metrics, how to prove that the weights of the regularised model have less variance (simpler) than the standard neural network?
  
  **Tags:** neural-networks, proofs, regularization, dropout, l1-regularization
  
  > **Comments:**
  > **username_1**: This question doesn't have an answer unless you define what you mean by complexity. If by complexity you mean the number of parameters, then both regularization methods you mentioned do nothing for complexity. If by complexity you mean the variance in parameters you might have more luck, but I imagine that dropout wouldn't show any reduced complexity.  
  > **username_0**: @username_1 Thank you I updated my question. But how does dropout not reduce complexity if it deactivates nodes during the training process? Having less nodes seems like a simpler model.  
  > **username_1**: It inconsistently deactivates nodes during training which means you need just as many parameters, and then at testing time it doesn't deactivate them at all  
  > **username_2**: Hello. I still think that this question is not clear enough, because of the part. "how to prove that the weights of the regularised model have less variance (simpler) than the standard neural network?". It seems that you're using "simpler" as a synonym for having less variance, but what do you mean by "less variance" here? You need to clarify this part, because, once we have 2 neural networks, we have 2 specific sets of weights, so the only thing that you can compute is really the variance of the values of these parameters.  
  > **username_2**: So, are you looking for a theoretical result that shows that, if using e.g. L2 regularization, your parameters will be limited to a smaller set of values, so will have less variance generally?  
  
  ---
  
  ## 0 answers
  
  