Answers.Body: <p>Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split for cross-validation.</p>
  <p>That means that any result metric from the neural network e.g. accuracy, loss, F1 score, is a random variable.</p>
  <p>Reporting a single value of a random variable is not very informative. It may be higher or lower than expected, and it is difficult to tell if a result is significant.</p>
  <p>Your reviewers are asking you to run the entire training and reporting from your dataset multiple times, to get the mean and standard deviation of any metrics that you reported.</p>
  <p>You can keep the same train/test split for your hold-out test data set. Ideally this is the same train/test split as used by the classifiers that you are comparing your thesis results with.</p>
  <p>If you used a seed for the RNG for repeatability, you can keep that as is and perform multiple training and reporting runs within a single script starting with the same seed set at the very beginning. Alternatively, if this would take too long, you could generate a set of seeds to use in consecutive runs - provided you are not selecting seed values depending on the results it does not matter.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-05T07:22:55.390Z
Answers.Text: Thank you so much Sir for your detailed answer. It helps me a lot.
Answers.UserDisplayName: Md. Asif Iqbal Fahim
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-05T07:20:19.647Z
Answers.LastActivityDate: 2021-07-05T07:20:19.647Z
Answers.UserDisplayName: Neil Slater
Body: <p>I got this feedback for my thesis paper.</p>
  <blockquote>
  <p>The improvement shown in the results section could be the result of random initialization. There should be multiple runs with means and standard deviations.</p>
  </blockquote>
  <p>Can anyone explain this feedback with details?</p>
  <p>I used a neural network with pre-trained weights for transfer learning (specifically, EfficientNetB0, with <code>'noisy-student'</code> for the <code>weights</code>). It was a classification problem to classify between Covid-19, Viral Pneumonia, and normal cases. I normalised the dataset so that the images are in the range [0, 255] and I also did k-fold cross-validation.</p>
  
Tags: <transfer-learning><weights-initialization><multiclass-classification><uncertainty-quantification><k-fold-cv>
Title: Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?
UserDisplayName: Md. Asif Iqbal Fahim
ThreadText: # Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?
  
  (Asked by: username_0 on 2021-07-05)
  
  I got this feedback for my thesis paper.
  
  > The improvement shown in the results section could be the result of random initialization. There should be multiple runs with means and standard deviations.
  
  Can anyone explain this feedback with details?
  
  I used a neural network with pre-trained weights for transfer learning (specifically, EfficientNetB0, with `'noisy-student'` for the `weights`). It was a classification problem to classify between Covid-19, Viral Pneumonia, and normal cases. I normalised the dataset so that the images are in the range [0, 255] and I also did k-fold cross-validation.
  
  **Tags:** transfer-learning, weights-initialization, multiclass-classification, uncertainty-quantification, k-fold-cv
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-05. Score: 3 | Accepted answer
  
  Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split for cross-validation.
  
  That means that any result metric from the neural network e.g. accuracy, loss, F1 score, is a random variable.
  
  Reporting a single value of a random variable is not very informative. It may be higher or lower than expected, and it is difficult to tell if a result is significant.
  
  Your reviewers are asking you to run the entire training and reporting from your dataset multiple times, to get the mean and standard deviation of any metrics that you reported.
  
  You can keep the same train/test split for your hold-out test data set. Ideally this is the same train/test split as used by the classifiers that you are comparing your thesis results with.
  
  If you used a seed for the RNG for repeatability, you can keep that as is and perform multiple training and reporting runs within a single script starting with the same seed set at the very beginning. Alternatively, if this would take too long, you could generate a set of seeds to use in consecutive runs - provided you are not selecting seed values depending on the results it does not matter.
  
  > **Comments:**
  > **username_0**: Thank you so much Sir for your detailed answer. It helps me a lot.  
  
  ---
  
  