Answers.Body: <blockquote>
    <p>I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?</p>
  </blockquote>
  
  <p>That is correct. If you fix the seed for a process which inherently has stochastic behaviour by design (such as initialising neural network params), then what you know about the model is that it is the best one given the hyperparameters you have selected and <em>that specific seed</em>. Sometimes the value of the seed is highly relevant, other times less so.</p>
  
  <blockquote>
    <p>Since results might be stochastic, how would researchers know what their best performing model is? </p>
  </blockquote>
  
  <p>In general, as with any experiment where measurements are variable, by running the experiment multiple times and taking statistics over the set of results. This will give you a much better sense of how the algorithm does in general, independently of specific seeds. You can still fix your RNG seeds for repeatability, but you will need multiple sets of them.</p>
  
  <p>For certain goals, such as making the best possible model that you can, independently of whether the approach you take is "best in general", this is not necessary. A single run which creates a state-of-the-art performance is still of interest, for instance. Or if you are creating a model that you want to use in production, you may care less about the stability of the technique (and being "lucky") than being in possession of a high performing agent for the task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-24T12:31:15.113Z
Answers.LastActivityDate: 2020-05-24T12:31:15.113Z
Answers.UserDisplayName: Neil Slater
Body: <p>There are many factors that cause the results of ML models to be different for every run of the same piece of code. One factor could be different initialization of weights in the neural network.</p>
  
  <p>Since results might be stochastic, how would researchers know what their best performing model is? I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-24T10:23:19.007Z
Comments.Text: Maybe it would be better that you list the parameters that introduce stochasticity in your training process, model, etc.
Comments.UserDisplayName: nbro
Tags: <deep-learning><reinforcement-learning><dqn>
Title: How would researchers determine the best deep learning model if every run of the code yields different results?
UserDisplayName: calveeen
ThreadText: # How would researchers determine the best deep learning model if every run of the code yields different results?
  
  (Asked by: username_0 on 2020-05-24)
  
  There are many factors that cause the results of ML models to be different for every run of the same piece of code. One factor could be different initialization of weights in the neural network.
  
  Since results might be stochastic, how would researchers know what their best performing model is? I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?
  
  **Tags:** deep-learning, reinforcement-learning, dqn
  > **Comments:**
  > **username_1**: Maybe it would be better that you list the parameters that introduce stochasticity in your training process, model, etc.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-24. Score: 3 | Accepted answer
  
  > I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results?
  
  That is correct. If you fix the seed for a process which inherently has stochastic behaviour by design (such as initialising neural network params), then what you know about the model is that it is the best one given the hyperparameters you have selected and *that specific seed*. Sometimes the value of the seed is highly relevant, other times less so.
  
  > Since results might be stochastic, how would researchers know what their best performing model is?
  
  In general, as with any experiment where measurements are variable, by running the experiment multiple times and taking statistics over the set of results. This will give you a much better sense of how the algorithm does in general, independently of specific seeds. You can still fix your RNG seeds for repeatability, but you will need multiple sets of them.
  
  For certain goals, such as making the best possible model that you can, independently of whether the approach you take is "best in general", this is not necessary. A single run which creates a state-of-the-art performance is still of interest, for instance. Or if you are creating a model that you want to use in production, you may care less about the stability of the technique (and being "lucky") than being in possession of a high performing agent for the task.
  
  
  ---
  
  