Answers.Body: <blockquote>
    <p>Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?</p>
  </blockquote>
  
  <p>Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):</p>
  
  <ul>
  <li>Neural network weight initialisation</li>
  <li>Action choice for <span class="math-container">$\epsilon$</span>-greedy or other behaviour policy (does not apply in your case, because you work exclusively from experience replay)</li>
  <li>Minibatch sampling from experience replay</li>
  <li>Stochastic choices in the environment (does not apply in your case)</li>
  <li>Other stochastic parts of training that may be in use, such as dropout regularisation</li>
  </ul>
  
  <p>Controlling all these should make your training process deterministic and repeatable. It won't necessarily make it correct.</p>
  
  <blockquote>
    <p>I reran the same script for the same <span class="math-container">$x$</span> number of epochs and got policy <span class="math-container">$\pi_2$</span>. I expected <span class="math-container">$\pi_1 $</span> and <span class="math-container">$\pi_2$</span> to be similar because i ran the same script.</p>
  </blockquote>
  
  <p>This is subtly different. It seems you hoped that convergence of the algorithm would mean you got to the same approximately optimal policy. In principle this is possible, because Q-learning should find a deterministic policy. However, there are some details to bear in mind:</p>
  
  <ul>
  <li><p>Many environments support multiple equivalent optimal policies. A simple grid world can have multiple equivalent paths from start to goal states. A Q-learning with approximation function will slightly prefer one or other path, resulting in very different, but still optimal, policies.</p></li>
  <li><p>Q-learning with approximation can go wrong and learn incorrectly. The usual checks and balances against this are running large numbers of simulations and testing. </p></li>
  </ul>
  
  <p>You don't have great options here, from your comments you are training purely offline from historic data. Your one sanity check - do I get the same policy if I re-try - has shown inconsistency. However, it doesn't necessarily mean you have a problem, perhaps the two policies are equivalent.</p>
  
  <p>Here are a couple of additional tests that may help:</p>
  
  <ul>
  <li><p>Instead of looking at the maximising action choice in the test data, look at how each Q function scores the behaviour policy action choice. If the scores are close (by some measure such as MSE), then the two Q-learners are basically agreeing and are more likely to have equivalent but different policies, as opposed to radically different end results.</p></li>
  <li><p>Have each Q network score the other's Q function action choice over an arbitrary (but realistic) set of states. If the values are similar to each other, then again this points to successful convergence given the training data, but with different outcomes due to small details.</p></li>
  </ul>
  
  <p>If either of these checks shows the networks are radically different, then you have a problem. Which run, if any, has found a viable policy, and which has failed? </p>
  
  <p>Even if the checks agree, it is circumstantial evidence that the Q learning process is stable, not proof that you have an agent that is better than the prevailing behaviour policy in your real world system.</p>
  
  <p>You won't know if the agent is truly better, unless you can find a more independent way to assess the agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T11:23:05.347Z
Answers.Text: Thanks Neil for the tips :-) I realised that it could be different because of prioritised replay sampling and initialisation of random normal weights. I wouldn't be able to believe if those random parameters affect the results :/
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T11:33:04.837Z
Answers.Text: can prioritised experience replay be made deterministic ? because the sampling is based on probabilities ?
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T11:34:46.817Z
Answers.Text: @calveen: Any "random" process in machine learning can be made deterministic by setting the RNG seeds. That won't necessarily make things better for your end result though. A Q function that is consistently inaccurate has limited use
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T12:49:03.873Z
Answers.Text: Just to clarify again, for the 1st test, we compute the q value for the action chosen by the behaviour policy at some state for each run of the script and we see whether the q values for that particular action are similar ?
  
  For the second test, we use the q network from 1st run to compute the q function for the action chosen by the q network from the second run over some states and vice versa. The q values output by these 2 networks should be similar ?
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T13:08:34.203Z
Answers.Text: @calveen: Yes. Note those tests I am suggesting are not formal parts of Q-learning theory. They are just metrics that may help you decide whether you have a problem with stability of your learning algorithm.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T15:11:45.257Z
Answers.Text: is there a particular reason why network weight initialisation might cause problems because the algorithm when it converges would have similar weights ? due to gradient descent update
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T16:08:10.093Z
Answers.Text: @calveeen: NN initialisation makes a huge difference, and even if the NN predicts the same values, if it was initialised differently the weights are likely to be very different. It is a major source of difference between training runs.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T10:37:14.853Z
Answers.LastActivityDate: 2020-05-21T13:16:02.927Z
Answers.LastEditDate: 2020-05-21T13:16:02.927Z
Answers.UserDisplayName: Neil Slater
Body: <p>I ran a deep q learning algorithm (DQN) for <span class="math-container">$x$</span> number of epochs and got policy <span class="math-container">$\pi_1$</span>. 
  I reran the same script for the same <span class="math-container">$x$</span> number of epochs and got policy <span class="math-container">$\pi_2$</span>. I expected <span class="math-container">$\pi_1 $</span> and <span class="math-container">$\pi_2$</span> to be similar because i ran the same script. However, when computing the actions on the same test set, i realised the actions were very different. </p>
  
  <p>Is this supposed to be normal when training deep q networks or is there something that I am missing ?</p>
  
  <p>I am using prioritised experience replay when training the model.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T08:52:35.290Z
Comments.Text: Have you measured the expected returns for the two different policies (separately from training runs)? If not, may be worth doing so and adding that detail to the question, as if the returns are similar it points to a different kind of answer to if they were really different
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T08:53:48.863Z
Comments.Text: Second clarifying question - are you calculating actions given arbitrary test states? Do those test states actually occur *naturally* when the agents behave optimally?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T08:55:29.467Z
Comments.Text: not yet because it is not straight forward to measure the value of the learnt policy because i would have to use off policy reinforcement learning evaluation techniques like Weighted importance sampling to do so.
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T08:58:29.983Z
Comments.Text: regarding second question, I have trajectories of a behaviour policy, and i split them into training and test sets. After training the Q network only on the train set, I feed the test set into the Q network and obtain the actions via an argmax for each state
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T09:42:53.187Z
Comments.Text: @calveen: Is all this because you have one real environment, that your agents are not allowed to act in, and no way to accurately simulate it?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T09:58:41.767Z
Comments.Text: I am implementing reinforcement learning on retrospective healthcare data to discover optimal treatment strategies. Hence it is difficult to simulate the learned target policy on actual patients.
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-21T10:03:47.383Z
Comments.Text: OK, So it would be hard to use weighted importance sampling in any case, because you don't have access to behaviour policy function $b(a|s)$ - and chances are that there are multiple policies in use in your training data. There are off-policy evaluation techniques that don't require importance sampling though. Q-learning used for policy evaluation is one of them.
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><reinforcement-learning><dqn>
Title: Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?
UserDisplayName: calveeen
ThreadText: # Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?
  
  (Asked by: username_0 on 2020-05-21)
  
  I ran a deep q learning algorithm (DQN) for $x$ number of epochs and got policy $\pi\_1$.
  I reran the same script for the same $x$ number of epochs and got policy $\pi\_2$. I expected $\pi\_1 $ and $\pi\_2$ to be similar because i ran the same script. However, when computing the actions on the same test set, i realised the actions were very different.
  
  Is this supposed to be normal when training deep q networks or is there something that I am missing ?
  
  I am using prioritised experience replay when training the model.
  
  **Tags:** machine-learning, reinforcement-learning, dqn
  
  > **Comments:**
  > **username_1**: Have you measured the expected returns for the two different policies (separately from training runs)? If not, may be worth doing so and adding that detail to the question, as if the returns are similar it points to a different kind of answer to if they were really different  
  > **username_1**: Second clarifying question - are you calculating actions given arbitrary test states? Do those test states actually occur *naturally* when the agents behave optimally?  
  > **username_0**: not yet because it is not straight forward to measure the value of the learnt policy because i would have to use off policy reinforcement learning evaluation techniques like Weighted importance sampling to do so.  
  > **username_0**: regarding second question, I have trajectories of a behaviour policy, and i split them into training and test sets. After training the Q network only on the train set, I feed the test set into the Q network and obtain the actions via an argmax for each state  
  > **username_1**: @calveen: Is all this because you have one real environment, that your agents are not allowed to act in, and no way to accurately simulate it?  
  > **username_0**: I am implementing reinforcement learning on retrospective healthcare data to discover optimal treatment strategies. Hence it is difficult to simulate the learned target policy on actual patients.  
  > **username_1**: OK, So it would be hard to use weighted importance sampling in any case, because you don't have access to behaviour policy function $b(a|s)$ - and chances are that there are multiple policies in use in your training data. There are off-policy evaluation techniques that don't require importance sampling though. Q-learning used for policy evaluation is one of them.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-21. Score: 3 | Accepted answer
  
  > Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?
  
  Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):
  
  * Neural network weight initialisation
  * Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, because you work exclusively from experience replay)
  * Minibatch sampling from experience replay
  * Stochastic choices in the environment (does not apply in your case)
  * Other stochastic parts of training that may be in use, such as dropout regularisation
  
  Controlling all these should make your training process deterministic and repeatable. It won't necessarily make it correct.
  
  > I reran the same script for the same $x$ number of epochs and got policy $\pi\_2$. I expected $\pi\_1 $ and $\pi\_2$ to be similar because i ran the same script.
  
  This is subtly different. It seems you hoped that convergence of the algorithm would mean you got to the same approximately optimal policy. In principle this is possible, because Q-learning should find a deterministic policy. However, there are some details to bear in mind:
  
  * Many environments support multiple equivalent optimal policies. A simple grid world can have multiple equivalent paths from start to goal states. A Q-learning with approximation function will slightly prefer one or other path, resulting in very different, but still optimal, policies.
  * Q-learning with approximation can go wrong and learn incorrectly. The usual checks and balances against this are running large numbers of simulations and testing.
  
  You don't have great options here, from your comments you are training purely offline from historic data. Your one sanity check - do I get the same policy if I re-try - has shown inconsistency. However, it doesn't necessarily mean you have a problem, perhaps the two policies are equivalent.
  
  Here are a couple of additional tests that may help:
  
  * Instead of looking at the maximising action choice in the test data, look at how each Q function scores the behaviour policy action choice. If the scores are close (by some measure such as MSE), then the two Q-learners are basically agreeing and are more likely to have equivalent but different policies, as opposed to radically different end results.
  * Have each Q network score the other's Q function action choice over an arbitrary (but realistic) set of states. If the values are similar to each other, then again this points to successful convergence given the training data, but with different outcomes due to small details.
  
  If either of these checks shows the networks are radically different, then you have a problem. Which run, if any, has found a viable policy, and which has failed?
  
  Even if the checks agree, it is circumstantial evidence that the Q learning process is stable, not proof that you have an agent that is better than the prevailing behaviour policy in your real world system.
  
  You won't know if the agent is truly better, unless you can find a more independent way to assess the agent.
  
  > **Comments:**
  > **username_0**: Thanks username_1 for the tips :-) I realised that it could be different because of prioritised replay sampling and initialisation of random normal weights. I wouldn't be able to believe if those random parameters affect the results :/  
  > **username_0**: can prioritised experience replay be made deterministic ? because the sampling is based on probabilities ?  
  > **username_1**: @calveen: Any "random" process in machine learning can be made deterministic by setting the RNG seeds. That won't necessarily make things better for your end result though. A Q function that is consistently inaccurate has limited use  
  > **username_0**: Just to clarify again, for the 1st test, we compute the q value for the action chosen by the behaviour policy at some state for each run of the script and we see whether the q values for that particular action are similar ?  For the second test, we use the q network from 1st run to compute the q function for the action chosen by the q network from the second run over some states and vice versa. The q values output by these 2 networks should be similar ?  
  > **username_1**: @calveen: Yes. Note those tests I am suggesting are not formal parts of Q-learning theory. They are just metrics that may help you decide whether you have a problem with stability of your learning algorithm.  
  > **username_0**: is there a particular reason why network weight initialisation might cause problems because the algorithm when it converges would have similar weights ? due to gradient descent update  
  > **username_1**: @username_0: NN initialisation makes a huge difference, and even if the NN predicts the same values, if it was initialised differently the weights are likely to be very different. It is a major source of difference between training runs.  
  
  ---
  
  