Answers.Body: <p>To my knowledge from reading about model-based and model-free reinforcement learning,</p>
  <p>DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):</p>
  <p><a href="https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1" rel="nofollow noreferrer">https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1</a></p>
  <p>You should see under the heading &quot;Definition&quot; (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: &quot;model-free techniques require extensive experience.&quot;</p>
  <p>Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.</p>
  <p>Also there are a lot of states: there are (<span class="math-container">$256^{128}$</span>) states. That is a really big number (I'm just emphasizing that training may take a long while)</p>
  <p>Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1509.06461.pdf</a> . I am assuming they used image data as well because Double DQN was compared to DQN.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-08T17:21:19.220Z
Answers.LastActivityDate: 2021-07-04T02:31:18.720Z
Answers.LastEditDate: 2021-07-04T02:31:18.720Z
Answers.UserDisplayName: rert588
Body: <p>I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used <em>double deep Q-learning</em>, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close. </p>
  
  <p>Did I miss something?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-21T16:37:40.990Z
Comments.Text: It does sound like something's off.  Have you already used aggregates of multiple frames so that the movement of the ball and paddle are captured?  My own solutions to this a few years back easily achieved perfect/infinite games after a few hours.
Comments.UserDisplayName: David Hoelzer
Tags: <reinforcement-learning><ai-design><q-learning><deep-rl>
Title: How to implement RAM versions of Atari games
UserDisplayName: dato nefaridze
ThreadText: # How to implement RAM versions of Atari games
  
  (Asked by: username_0 on 2020-06-06)
  
  I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used *double deep Q-learning*, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close.
  
  Did I miss something?
  
  **Tags:** reinforcement-learning, ai-design, q-learning, deep-rl
  > **Comments:**
  > **username_1**: It does sound like something's off.  Have you already used aggregates of multiple frames so that the movement of the ball and paddle are captured?  My own solutions to this a few years back easily achieved perfect/infinite games after a few hours.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-08. Score: 0
  
  To my knowledge from reading about model-based and model-free reinforcement learning,
  
  DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):
  
  <https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1>
  
  You should see under the heading "Definition" (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: "model-free techniques require extensive experience."
  
  Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.
  
  Also there are a lot of states: there are ($256^{128}$) states. That is a really big number (I'm just emphasizing that training may take a long while)
  
  Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: <https://arxiv.org/pdf/1509.06461.pdf> . I am assuming they used image data as well because Double DQN was compared to DQN.
  
  
  ---
  
  