Body: <p>Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the training itself?</p>
  <p>I mean maybe something like how deep they are, amount of conv/pooling/fully connected layers, size of filters, size of steps of filters, etc. any pointers that would help with regularisation.</p>
  <p><strong>EDIT</strong>: To explain deeper what I mean I might add that I am exploring an experimental idea for the training of the CNNs that is not in any way related to typical gradient descent with backpropagation. That is why typical methods related to training will not work. I can see already that the models train satisfactorily on the training set but don't perform that well on a test set and since I didn't figure out any regularization methods for this type of training I thought maybe there are some related to architecture, that the training process will have to abide.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-16T10:55:31.750Z
Comments.Text: I am not sure I understand your question. If you use e.g. L2 regularization, you're limiting the values that the weights can take, so, in this way, this type of regularization is related to the "architecture" of the neural network. So, can you clarify what you're really asking and why regularization techniques like "L2" or "dropout" are not the answers to your question?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-17T13:52:55.693Z
Comments.Text: Without more info about your training procedure, it's still not clear (at least to me),  why dropout does not work in your case. Have you already tried it?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-18T08:25:49.880Z
Comments.Text: @nbro The training is based on evolutionary algorithms with a very specific domain based fitness function without use of standard error metrics such as binary crossentropy or MSE. I can't even imagine where would dropout or L2 go into this process.
Comments.UserDisplayName: GKozinski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-26T20:11:26.543Z
Comments.Text: maxnorm over weight vectors is a regularisation technique that is not architectural, but may help in your situation
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><deep-learning><convolutional-neural-networks><architecture><regularization>
Title: Are there regularisation methods related only to architecture of the CNNs?
UserDisplayName: GKozinski
ThreadText: # Are there regularisation methods related only to architecture of the CNNs?
  
  (Asked by: username_0 on 2021-03-15)
  
  Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the training itself?
  
  I mean maybe something like how deep they are, amount of conv/pooling/fully connected layers, size of filters, size of steps of filters, etc. any pointers that would help with regularisation.
  
  **EDIT**: To explain deeper what I mean I might add that I am exploring an experimental idea for the training of the CNNs that is not in any way related to typical gradient descent with backpropagation. That is why typical methods related to training will not work. I can see already that the models train satisfactorily on the training set but don't perform that well on a test set and since I didn't figure out any regularization methods for this type of training I thought maybe there are some related to architecture, that the training process will have to abide.
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, architecture, regularization
  > **Comments:**
  > **username_1**: I am not sure I understand your question. If you use e.g. L2 regularization, you're limiting the values that the weights can take, so, in this way, this type of regularization is related to the "architecture" of the neural network. So, can you clarify what you're really asking and why regularization techniques like "L2" or "dropout" are not the answers to your question?  
  > **username_1**: Without more info about your training procedure, it's still not clear (at least to me),  why dropout does not work in your case. Have you already tried it?  
  > **username_0**: @nbro The training is based on evolutionary algorithms with a very specific domain based fitness function without use of standard error metrics such as binary crossentropy or MSE. I can't even imagine where would dropout or L2 go into this process.  
  > **username_2**: maxnorm over weight vectors is a regularisation technique that is not architectural, but may help in your situation  
  
  ---
  
  ## 0 answers
  
  