Body: <p>I'm using Experience Replay based on the original <a href="https://arxiv.org/pdf/1511.05952.pdf" rel="nofollow noreferrer">Prioritized Experience Replay (PER)</a> paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. There is space for further improvement, since PER remembers all experiences, regardless of their importance.</p>
  <p>I'd like to extend PER so it remembers selectively based on some <strong>metric</strong>, which would determine whether the experience is worth remembering or not. The time of sampling and re-adjusting the importance of the experiences increases with the number of samples remembered, so being smart about remembering should at the very least speed-up the replay, and hopefully also show some increase in data efficiency.</p>
  <p>Important design constrains for this <strong>remembering metric</strong>:</p>
  <ul>
  <li>compatibility with Q-Learning, such as DQN</li>
  <li>computation time, to speed up the process of learning and not trade off one type of computation for another</li>
  <li>simplicity</li>
  </ul>
  <hr />
  <p>My questions:</p>
  <ol>
  <li>What considerations would you make for designing such a metric?</li>
  <li>Do you know about any articles addressing the prioritized experience memorization for Q-Learning?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-18T11:54:58.083Z
Comments.Text: I recently saw a paper on arXiv which, based on a very quick scan of title/abstract/introduction, seems to be very closely related to what you're asking about: "Double Prioritized State Recycled Experience Replay" (https://arxiv.org/abs/2007.03961). Unforutnately I didn't get around to reading it in detail yet, so I can't effectively summarise it in a proper answer to your question. If you do indeed find useful stuff in there, it would be nice if you could write an answer yourself to your own question (for future visitors of the site).
Comments.UserDisplayName: Dennis Soemers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-21T07:26:04.030Z
Comments.Text: Thanks Dennis, appreciated. I read the article, but I don't fully trust its results. There are quite a few spelling errors in the writing and ambiguities in reporting results.
Comments.UserDisplayName: conscious_process
Tags: <reinforcement-learning><q-learning><dqn><experience-replay>
Title: Prioritised Remembering in Experience Replay (Q-Learning)
UserDisplayName: conscious_process
ThreadText: # Prioritised Remembering in Experience Replay (Q-Learning)
  
  (Asked by: username_0 on 2020-07-17)
  
  I'm using Experience Replay based on the original [Prioritized Experience Replay (PER)](https://arxiv.org/pdf/1511.05952.pdf) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. There is space for further improvement, since PER remembers all experiences, regardless of their importance.
  
  I'd like to extend PER so it remembers selectively based on some **metric**, which would determine whether the experience is worth remembering or not. The time of sampling and re-adjusting the importance of the experiences increases with the number of samples remembered, so being smart about remembering should at the very least speed-up the replay, and hopefully also show some increase in data efficiency.
  
  Important design constrains for this **remembering metric**:
  
  * compatibility with Q-Learning, such as DQN
  * computation time, to speed up the process of learning and not trade off one type of computation for another
  * simplicity
  
  ---
  
  My questions:
  
  1. What considerations would you make for designing such a metric?
  2. Do you know about any articles addressing the prioritized experience memorization for Q-Learning?
  
  **Tags:** reinforcement-learning, q-learning, dqn, experience-replay
  > **Comments:**
  > **username_1**: I recently saw a paper on arXiv which, based on a very quick scan of title/abstract/introduction, seems to be very closely related to what you're asking about: "Double Prioritized State Recycled Experience Replay" (https://arxiv.org/abs/2007.03961). Unforutnately I didn't get around to reading it in detail yet, so I can't effectively summarise it in a proper answer to your question. If you do indeed find useful stuff in there, it would be nice if you could write an answer yourself to your own question (for future visitors of the site).  
  > **username_0**: Thanks Dennis, appreciated. I read the article, but I don't fully trust its results. There are quite a few spelling errors in the writing and ambiguities in reporting results.  
  
  ---
  
  ## 0 answers
  
  