Answers.Body: <p>More specifically than off-policy RL, you are looking at <strong>offline</strong> reinforcement learning techniques. In offline RL, all training data is known beforehand (stationary), which is in stark contrast to the usual non-stationarity of online RL data.</p>
  <blockquote>
  <p>In my case, I already have all the experience</p>
  </blockquote>
  <p>Are you absolutely sure of this? If your data doesn't contain transitions from high-reward regions of the state space, isn't generalizable to new situations, or is from very disparate and unconnected regions of the state space, then your agent could have trouble learning a good policy, let alone an optimal policy (see &quot;When and why does offline RL work?&quot; slide from this <a href="https://sites.google.com/view/offlinerltutorial-neurips2020/home" rel="nofollow noreferrer">tutorial</a> by Kumar and Levine).</p>
  <p>If your data is indeed a good fit for offline RL, then there are certainly different updates that are recommended. In the same tutorial linked previously, the authors display some updates that are more conservative than traditional off-policy methods, especially when the data has high uncertainty. To answer your question - since you are currently using Q-learning, I suggest that you look into the <em>conservative Q-learning</em> (CQL) update as presented in the tutorial. To keep this answer brief and since I don't have much experience in this field, I will refer you to the corresponding <a href="https://arxiv.org/pdf/2005.01643.pdf" rel="nofollow noreferrer">survey paper</a> for additional understanding and other possible updates.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T09:22:14.060Z
Answers.Text: Thanks for your reply. I also looked into offline RL now, whose leading representative and researchers seems to be Sergey Levine. I'll check out the video as well, thanks. Conservative Q-Learning seems like a promising model to try out. And I would say yes the sampled experience should (hopefully) cover some high reward areas to learn from (millions of s,a,r,s' pairs). And there is no other option to generate more experience (too expensive/risky). So goal is now to squeeze out the best possible policy given the sampled experience.
Answers.UserDisplayName: PatrickSVM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-03T18:35:44.280Z
Answers.LastActivityDate: 2023-03-03T18:35:44.280Z
Answers.UserDisplayName: DeepQZero
Body: <p>I want to train a DQN model in an off-policy fashion, where my behavior policy is an older agent. I have a big memory of a lot of episodes of this agent. Now I want to find a better policy using DQN. Now I am just wondering, in the &quot;normal&quot; DQN case you would use the experience replay buffer and would update behavior and target policy online (behavior not really online but with the time lag introduced after which these parameters are also updated).</p>
  <p>In my case, I already have all the experience and would like to learn from it. Do you think it makes sense to use the exact same procedure in this context, so sampling one new action, state and immediate reward, follow up action or could it be better here to use the fact that all experience is already stored to exchange <span class="math-container">$R_{t+1} + \gamma  \max Q(S_{t+1},a)$</span> with some more future information about the rewards (up to the point of Monte Carlo where you take <span class="math-container">$G_t$</span> so the discounted cumulative reward seen during the episode from point t onwards)?</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><experience-replay><offline-reinforcement-learning>
Title: DQN with experience history to learn from already saved - which reward should I take?
UserDisplayName: PatrickSVM
ThreadText: # DQN with experience history to learn from already saved - which reward should I take?
  
  (Asked by: username_0 on 2023-01-26)
  
  I want to train a DQN model in an off-policy fashion, where my behavior policy is an older agent. I have a big memory of a lot of episodes of this agent. Now I want to find a better policy using DQN. Now I am just wondering, in the "normal" DQN case you would use the experience replay buffer and would update behavior and target policy online (behavior not really online but with the time lag introduced after which these parameters are also updated).
  
  In my case, I already have all the experience and would like to learn from it. Do you think it makes sense to use the exact same procedure in this context, so sampling one new action, state and immediate reward, follow up action or could it be better here to use the fact that all experience is already stored to exchange $R\_{t+1} + \gamma \max Q(S\_{t+1},a)$ with some more future information about the rewards (up to the point of Monte Carlo where you take $G\_t$ so the discounted cumulative reward seen during the episode from point t onwards)?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, experience-replay, offline-reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-03. Score: 1
  
  More specifically than off-policy RL, you are looking at **offline** reinforcement learning techniques. In offline RL, all training data is known beforehand (stationary), which is in stark contrast to the usual non-stationarity of online RL data.
  
  > In my case, I already have all the experience
  
  Are you absolutely sure of this? If your data doesn't contain transitions from high-reward regions of the state space, isn't generalizable to new situations, or is from very disparate and unconnected regions of the state space, then your agent could have trouble learning a good policy, let alone an optimal policy (see "When and why does offline RL work?" slide from this [tutorial](https://sites.google.com/view/offlinerltutorial-neurips2020/home) by Kumar and Levine).
  
  If your data is indeed a good fit for offline RL, then there are certainly different updates that are recommended. In the same tutorial linked previously, the authors display some updates that are more conservative than traditional off-policy methods, especially when the data has high uncertainty. To answer your question - since you are currently using Q-learning, I suggest that you look into the *conservative Q-learning* (CQL) update as presented in the tutorial. To keep this answer brief and since I don't have much experience in this field, I will refer you to the corresponding [survey paper](https://arxiv.org/pdf/2005.01643.pdf) for additional understanding and other possible updates.
  
  > **Comments:**
  > **username_0**: Thanks for your reply. I also looked into offline RL now, whose leading representative and researchers seems to be Sergey Levine. I'll check out the video as well, thanks. Conservative Q-Learning seems like a promising model to try out. And I would say yes the sampled experience should (hopefully) cover some high reward areas to learn from (millions of s,a,r,s' pairs). And there is no other option to generate more experience (too expensive/risky). So goal is now to squeeze out the best possible policy given the sampled experience.  
  
  ---
  
  