Body: <p>I have two sets of data, one training and one test set. I use the train set to train the deep q network model variant. I also continuously evaluate the agent Q values obtained on the test set every 5000 epochs I find that the agent Q values on the test set do not converge and neither do the policies.</p>
  
  <p>iteration <span class="math-container">$x$</span>: Q values for the first 5 test data are <code>[15.271439, 13.013742, 14.137051, 13.96463, 11.490129]</code> with policies: <code>[15, 0, 0, 0, 15]</code></p>
  
  <p>iteration <span class="math-container">$x+10000$</span>:
  Q values for the first 5 test data are <code>[15.047309, 15.5233555, 16.786497, 16.100864, 13.066223]</code> with policies: <code>[0, 0, 0, 0, 15]</code></p>
  
  <p>This means that the weights of the neural network are not converging. Although I can manually test each policy at each iteration and decide which of the policy performs best, I would like to know if correct training of the network would lead to weight convergence ?</p>
  
  <p>Training loss plot:
  <a href="https://i.sstatic.net/ocSIG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ocSIG.png" alt="enter image description here"></a></p>
  
  <p>You can see that the loss decreases over time however, there are occasional spikes in the loss which does not seem to go away.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T09:10:39.283Z
Comments.Text: if you already have the data, why are you using rl? would this not be a supervised/unsupervised problem?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T09:49:17.693Z
Comments.Text: It is not a supervised learning problem. The train set contains experiences and can be seen as the "replay buffer", just that the no new experiences can be collected. As I cannot obtain the "rewards" for the current policy in real time, I am evaluating the trained DQN on the "replay buffer" and analysing the Q scores that the neural network would output on unseen states (test data) for every few cycles of epochs
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T10:10:10.977Z
Comments.Text: how do you generate the experience? will the experience you store not be from a really poor policy since you're not optimising at the same time? it sounds like you would have to gather experience, train the dqn a bit to improve the policy, then get more experience, otherwise you will just continue to get experience from a poor policy and the dqn could never get close to the optimal solution.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T10:12:18.567Z
Comments.Text: so because I am trying to apply reinforcement learning techniques on clinical data, it would not be possible for me to gather new experiences and I would have to work with the current collected experiences demonstrated by the clinicans.
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T10:22:13.693Z
Comments.Text: what type of clinical data is it? are you trying to figure out the best treatment group?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T10:26:15.907Z
Comments.Text: I’m trying to figure out the best treatment policies for sepsis patients and patient states are continuous variables, hence the DQN is needed. The actions (drug doses) are discretised into different bins.
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T12:11:28.083Z
Comments.Text: What does the loss look like when you are training the DQN? Is it decreasing over time?
Comments.UserDisplayName: user5093249
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T12:55:01.177Z
Comments.Text: @user5093249 the loss decreases over time, but there are still occasional spikes that are difficult to get rid off despite training for longer periods of time. The deep Q network uses a prioritised experience replay together with some regularisation. I have linked the paper that I have adapted the model from here. https://arxiv.org/abs/1711.09602. Are the Q values being learnt well if it does not converge ?
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T14:48:49.913Z
Comments.Text: Thanks @calveeen. I also came across that paper while trying to understand how DQN fits in your use case. Re- the plot, the fact that it has a general decreasing trend seems good. The spikes may be due to the instants where you update the target network (or other reasons see [this](https://stats.stackexchange.com/questions/303857/explanation-of-spikes-in-training-loss-vs-iterations-with-adam-optimizer) or [this](https://stackoverflow.com/questions/47824598/why-does-my-training-loss-have-regular-spikes)).
Comments.UserDisplayName: user5093249
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T14:59:52.080Z
Comments.Text: I would expect the Q values of the agent's chosen action to converge for the test data though as this would entail that the network weights have converged. However the policies obtained do not seem to resemble each other (I do a evaluation on the test set every 10,000 epochs). This makes it very difficult to know whether the optimal policy has truly been found
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T15:15:14.213Z
Comments.Text: It is not very clear to me how you use your test data to evaluate the DQN. Doesn’t the test data contain the actions already taken by the physician? Do you replace those by the actions returned by the trained DQN?
  Re- “This makes it very difficult to know whether the optimal policy has truly been found”, I agree with this answer [here](https://ai.stackexchange.com/questions/21182/how-to-evaluate-a-deep-q-network) that it would be better to evaluate the training performance of your DQN based on the average reward it is getting instead of relying on the Q-values.
Comments.UserDisplayName: user5093249
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T15:20:14.657Z
Comments.Text: Yes the test data contains already the actions taken by physicians. What I am doing is that I am feeding the states in the test set into the trained DQN, I then analyse the agent's Q values and actions (based on greedy policy) and I notice that the Q values of the argmax action does not converge, and the actions of the agent likewise do not seem to approach an optimal policy. I provided a sample of what the Q values look like when this evaluation was carried out at different time steps.
Comments.UserDisplayName: calveeen
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-09T15:37:40.077Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/109103/discussion-between-user5093249-and-calveeen).
Comments.UserDisplayName: user5093249
Tags: <deep-learning><reinforcement-learning><dqn>
Title: Should the network weights converge when training Deep Q networks?
UserDisplayName: calveeen
ThreadText: # Should the network weights converge when training Deep Q networks?
  
  (Asked by: username_0 on 2020-06-09)
  
  I have two sets of data, one training and one test set. I use the train set to train the deep q network model variant. I also continuously evaluate the agent Q values obtained on the test set every 5000 epochs I find that the agent Q values on the test set do not converge and neither do the policies.
  
  iteration $x$: Q values for the first 5 test data are `[15.271439, 13.013742, 14.137051, 13.96463, 11.490129]` with policies: `[15, 0, 0, 0, 15]`
  
  iteration $x+10000$:
  Q values for the first 5 test data are `[15.047309, 15.5233555, 16.786497, 16.100864, 13.066223]` with policies: `[0, 0, 0, 0, 15]`
  
  This means that the weights of the neural network are not converging. Although I can manually test each policy at each iteration and decide which of the policy performs best, I would like to know if correct training of the network would lead to weight convergence ?
  
  Training loss plot:
  [![enter image description here](https://i.sstatic.net/ocSIG.png)](https://i.sstatic.net/ocSIG.png)
  
  You can see that the loss decreases over time however, there are occasional spikes in the loss which does not seem to go away.
  
  **Tags:** deep-learning, reinforcement-learning, dqn
  > **Comments:**
  > **username_1**: if you already have the data, why are you using rl? would this not be a supervised/unsupervised problem?  
  > **username_0**: It is not a supervised learning problem. The train set contains experiences and can be seen as the "replay buffer", just that the no new experiences can be collected. As I cannot obtain the "rewards" for the current policy in real time, I am evaluating the trained DQN on the "replay buffer" and analysing the Q scores that the neural network would output on unseen states (test data) for every few cycles of epochs  
  > **username_1**: how do you generate the experience? will the experience you store not be from a really poor policy since you're not optimising at the same time? it sounds like you would have to gather experience, train the dqn a bit to improve the policy, then get more experience, otherwise you will just continue to get experience from a poor policy and the dqn could never get close to the optimal solution.  
  > **username_0**: so because I am trying to apply reinforcement learning techniques on clinical data, it would not be possible for me to gather new experiences and I would have to work with the current collected experiences demonstrated by the clinicans.  
  > **username_1**: what type of clinical data is it? are you trying to figure out the best treatment group?  
  > **username_0**: I’m trying to figure out the best treatment policies for sepsis patients and patient states are continuous variables, hence the DQN is needed. The actions (drug doses) are discretised into different bins.  
  > **username_2**: What does the loss look like when you are training the DQN? Is it decreasing over time?  
  > **username_0**: @user5093249 the loss decreases over time, but there are still occasional spikes that are difficult to get rid off despite training for longer periods of time. The deep Q network uses a prioritised experience replay together with some regularisation. I have linked the paper that I have adapted the model from here. https://arxiv.org/abs/1711.09602. Are the Q values being learnt well if it does not converge ?  
  > **username_2**: Thanks @calveeen. I also came across that paper while trying to understand how DQN fits in your use case. Re- the plot, the fact that it has a general decreasing trend seems good. The spikes may be due to the instants where you update the target network (or other reasons see [this](https://stats.stackexchange.com/questions/303857/explanation-of-spikes-in-training-loss-vs-iterations-with-adam-optimizer) or [this](https://stackoverflow.com/questions/47824598/why-does-my-training-loss-have-regular-spikes)).  
  > **username_0**: I would expect the Q values of the agent's chosen action to converge for the test data though as this would entail that the network weights have converged. However the policies obtained do not seem to resemble each other (I do a evaluation on the test set every 10,000 epochs). This makes it very difficult to know whether the optimal policy has truly been found  
  > **username_2**: It is not very clear to me how you use your test data to evaluate the DQN. Doesn’t the test data contain the actions already taken by the physician? Do you replace those by the actions returned by the trained DQN? Re- “This makes it very difficult to know whether the optimal policy has truly been found”, I agree with this answer [here](https://ai.stackexchange.com/questions/21182/how-to-evaluate-a-deep-q-network) that it would be better to evaluate the training performance of your DQN based on the average reward it is getting instead of relying on the Q-values.  
  > **username_0**: Yes the test data contains already the actions taken by physicians. What I am doing is that I am feeding the states in the test set into the trained DQN, I then analyse the agent's Q values and actions (based on greedy policy) and I notice that the Q values of the argmax action does not converge, and the actions of the agent likewise do not seem to approach an optimal policy. I provided a sample of what the Q values look like when this evaluation was carried out at different time steps.  
  > **username_2**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/109103/discussion-between-user5093249-and-calveeen).  
  
  ---
  
  ## 0 answers
  
  