Body: <p>I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc.</p>
  
  <p>A neural network is not a deterministic algorithm, so, in each training set, I train the neural network from scratch and I stop it when it's full converged.</p>
  
  <p>After training is complete, I calculate the performance of the neural network in a test dataset. The problem is, I trained the neural network from scratch 2 times with the same parameters, but the difference in performance was almost 5%, which is a BIG DIFFERENCE.</p>
  
  <p>So, what's the reasonable number of training runs to obtain a credible performance number of a neural network?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T12:44:37.907Z
Comments.Text: Are you shuffling the data before passing it to the net? Shuffling the data can cause these "issues" because that operation introduces randomness.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T13:05:10.590Z
Comments.Text: No, I'm not shuffling. My neural network is a CNN and in each mini-batch i pad all images to have same size. So, all images in same mini batch has the size of the biggest image in that mini batch. 
  But if i'm not shuffling, should not be the same in the 2 runs?
Comments.UserDisplayName: João Castilho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T13:10:25.597Z
Comments.Text: Are you using the same test set for both experiments? Also, have you tried to initialize the weights with the same values (not with the same strategy)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T13:12:38.940Z
Comments.Text: Yes, the trainning,validation and test sets are always the same. I'm initialize the weights with Glorot Uniform with default parameters
Comments.UserDisplayName: João Castilho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T13:15:07.717Z
Comments.Text: Initializing the weights with Glorot uniform may create a different set of initial weights. So, you may want to try initializing both neural networks with the _exact_ same weights, to see if that solves the issue.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T16:21:29.093Z
Comments.Text: And if I want to test the use of Glorot Uniform initialization in my neural network? How many runs It's reasonable to do?
Comments.UserDisplayName: João Castilho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T16:23:01.250Z
Comments.Text: Is it normal doing more than 1 run for gain credibility? I know in genetic algorithms, it's usual to run at least 30 runs, but I never saw nothing about this is Neural Networks
Comments.UserDisplayName: João Castilho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T16:32:25.210Z
Comments.Text: Nitpick: "A neural network is not a deterministic algorithm". Stochasticity comes from the samples and weight initialization. Everything else is deterministic. Plus neural network is not an algorithm but I guess I know what you mean.
Comments.UserDisplayName: nyumerics
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-06T20:07:29.920Z
Comments.Text: @JoãoCastilho If you remove all sources of randomness, the outputs of the networks and their parameters should be exactly the same, if they are trained in the same way and for the exact same number of iterations (epochs and steps). If you introduce randomness, then, yes, you may want to train different networks to gain a sense of confidence about the network and its output.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-07T09:55:47.440Z
Comments.Text: Please add a seed value in your train/test split code and weight Initialization. Though 5% is too much to be attributed to randomness.
Comments.UserDisplayName: 10xAI
Tags: <neural-networks><training><optimization><hyperparameter-optimization><performance>
Title: How many training runs are needed to obtain a credible value for performance?
UserDisplayName: João Castilho
ThreadText: # How many training runs are needed to obtain a credible value for performance?
  
  (Asked by: username_0 on 2020-06-06)
  
  I'm trying to optimize a neural network. For that, I'm changing parameters like the batch size, learning rate, weight initialization, etc.
  
  A neural network is not a deterministic algorithm, so, in each training set, I train the neural network from scratch and I stop it when it's full converged.
  
  After training is complete, I calculate the performance of the neural network in a test dataset. The problem is, I trained the neural network from scratch 2 times with the same parameters, but the difference in performance was almost 5%, which is a BIG DIFFERENCE.
  
  So, what's the reasonable number of training runs to obtain a credible performance number of a neural network?
  
  **Tags:** neural-networks, training, optimization, hyperparameter-optimization, performance
  
  > **Comments:**
  > **username_1**: Are you shuffling the data before passing it to the net? Shuffling the data can cause these "issues" because that operation introduces randomness.  
  > **username_0**: No, I'm not shuffling. My neural network is a CNN and in each mini-batch i pad all images to have same size. So, all images in same mini batch has the size of the biggest image in that mini batch.  But if i'm not shuffling, should not be the same in the 2 runs?  
  > **username_1**: Are you using the same test set for both experiments? Also, have you tried to initialize the weights with the same values (not with the same strategy)?  
  > **username_0**: Yes, the trainning,validation and test sets are always the same. I'm initialize the weights with Glorot Uniform with default parameters  
  > **username_1**: Initializing the weights with Glorot uniform may create a different set of initial weights. So, you may want to try initializing both neural networks with the _exact_ same weights, to see if that solves the issue.  
  > **username_0**: And if I want to test the use of Glorot Uniform initialization in my neural network? How many runs It's reasonable to do?  
  > **username_0**: Is it normal doing more than 1 run for gain credibility? I know in genetic algorithms, it's usual to run at least 30 runs, but I never saw nothing about this is Neural Networks  
  > **username_2**: Nitpick: "A neural network is not a deterministic algorithm". Stochasticity comes from the samples and weight initialization. Everything else is deterministic. Plus neural network is not an algorithm but I guess I know what you mean.  
  > **username_1**: @username_0 If you remove all sources of randomness, the outputs of the networks and their parameters should be exactly the same, if they are trained in the same way and for the exact same number of iterations (epochs and steps). If you introduce randomness, then, yes, you may want to train different networks to gain a sense of confidence about the network and its output.  
  > **username_3**: Please add a seed value in your train/test split code and weight Initialization. Though 5% is too much to be attributed to randomness.  
  
  ---
  
  ## 0 answers
  
  