Answers.Body: <p>I don't think you can.</p>
  
  <p>Say a NN with 3 layers gives an accuracy of 95.3% and another NN with 4 layers gives an accuracy of 95.4%. Then there is no guarantee that the 4 layer NN is better than the 3 layered NN. Since with different initial values the 3 layer NN might perform better.</p>
  
  <p>You could run multiple times and probabilistically say that this is better, but this is computational intense.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-11T13:21:35.440Z
Answers.LastActivityDate: 2020-01-11T13:21:35.440Z
Answers.UserDisplayName: codeblooded
Answers.Body: <p>There are other sources that will lead to different results in addition to weight initialization. For example dropout layers. Make sure you specify the random seed.Also data reading using flow from directory,make sure you set shuffle to False or if you do not then set the random seed. If you use transfer learning make that part of your network non-trainable. Some networks have dropout in them and do not provide a way to set the random seed. IF you are using a GPU there are even more issues to contend with.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-21T16:20:27.363Z
Answers.LastActivityDate: 2020-03-21T16:20:27.363Z
Answers.UserDisplayName: Gerry P
Answers.Body: <p>There are two weight-initializing methods for neural networks: 
  1-Zero initializing 
  2-Random initializing </p>
  
  <p><a href="https://towardsdatascience.com/weight-initialization-techniques-in-neural-networks-26c649eb3b78" rel="nofollow noreferrer">https://towardsdatascience.com/weight-initialization-techniques-in-neural-networks-26c649eb3b78</a></p>
  
  <p>If you choose zero initalizing method in every train loop, you may get same results OR you can use transfer learning according to your problem, it allows to start same parameters. At last, as a worst and hardest choice, you can write your own weight arrays and feed your layers.</p>
  
  <p>Problem you mentioned is one of the most interesting problems in the evaluation of performance of neural networks. You can use cross-validation method to verify your model's accuracy! ıt will give more reliable results!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-11T23:58:36.993Z
Answers.LastActivityDate: 2020-01-11T23:58:36.993Z
Answers.UserDisplayName: Barış Akın
Body: <p>In the step of tuning my neural networks I often encounter a problem that every time I train the exact same network, it gives me different final error due to random initialization of the weights. Sometimes the differences are small and negligible, sometimes they are significant, depending on the data and architecture.</p>
  
  <p>My problem arises when I want to tune some parameters like number of layers or neurons, because I don't know if the change in final error was caused by recent changes in network's architecture or it is simply effect of the aforementioned randomness. </p>
  
  <p>My question is how to deal with this issue?</p>
  
Tags: <neural-networks><training><randomness>
Title: How to deal with random weights initialization in hyperparameters tuning?
UserDisplayName: GKozinski
ThreadText: # How to deal with random weights initialization in hyperparameters tuning?
  
  (Asked by: username_0 on 2020-01-11)
  
  In the step of tuning my neural networks I often encounter a problem that every time I train the exact same network, it gives me different final error due to random initialization of the weights. Sometimes the differences are small and negligible, sometimes they are significant, depending on the data and architecture.
  
  My problem arises when I want to tune some parameters like number of layers or neurons, because I don't know if the change in final error was caused by recent changes in network's architecture or it is simply effect of the aforementioned randomness.
  
  My question is how to deal with this issue?
  
  **Tags:** neural-networks, training, randomness
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-01-11. Score: 2
  
  I don't think you can.
  
  Say a NN with 3 layers gives an accuracy of 95.3% and another NN with 4 layers gives an accuracy of 95.4%. Then there is no guarantee that the 4 layer NN is better than the 3 layered NN. Since with different initial values the 3 layer NN might perform better.
  
  You could run multiple times and probabilistically say that this is better, but this is computational intense.
  
  
  ---
  
  ### Answer by username_2 on 2020-03-21. Score: 1
  
  There are other sources that will lead to different results in addition to weight initialization. For example dropout layers. Make sure you specify the random seed.Also data reading using flow from directory,make sure you set shuffle to False or if you do not then set the random seed. If you use transfer learning make that part of your network non-trainable. Some networks have dropout in them and do not provide a way to set the random seed. IF you are using a GPU there are even more issues to contend with.
  
  
  ---
  
  ### Answer by username_3 on 2020-01-11. Score: 1
  
  There are two weight-initializing methods for neural networks:
  1-Zero initializing
  2-Random initializing
  
  <https://towardsdatascience.com/weight-initialization-techniques-in-neural-networks-26c649eb3b78>
  
  If you choose zero initalizing method in every train loop, you may get same results OR you can use transfer learning according to your problem, it allows to start same parameters. At last, as a worst and hardest choice, you can write your own weight arrays and feed your layers.
  
  Problem you mentioned is one of the most interesting problems in the evaluation of performance of neural networks. You can use cross-validation method to verify your model's accuracy! ıt will give more reliable results!
  
  
  ---
  
  