<p>To my knowledge from reading about model-based and model-free reinforcement learning,</p>
<p>DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):</p>
<p><a href="https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1" rel="nofollow noreferrer">https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1</a></p>
<p>You should see under the heading &quot;Definition&quot; (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: &quot;model-free techniques require extensive experience.&quot;</p>
<p>Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.</p>
<p>Also there are a lot of states: there are (<span class="math-container">$256^{128}$</span>) states. That is a really big number (I'm just emphasizing that training may take a long while)</p>
<p>Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1509.06461.pdf</a> . I am assuming they used image data as well because Double DQN was compared to DQN.</p>

CC BY-SA 4.0
2020-06-08T17:21:19.220Z
2021-07-04T02:31:18.720Z
2021-07-04T02:31:18.720Z
rert588
<p>I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used <em>double deep Q-learning</em>, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close. </p>

<p>Did I miss something?</p>

CC BY-SA 4.0
2023-11-21T16:37:40.990Z
It does sound like something's off.  Have you already used aggregates of multiple frames so that the movement of the ball and paddle are captured?  My own solutions to this a few years back easily achieved perfect/infinite games after a few hours.
David Hoelzer
<reinforcement-learning><ai-design><q-learning><deep-rl>
How to implement RAM versions of Atari games
dato nefaridze
# How to implement RAM versions of Atari games

(Asked by: username_0 on 2020-06-06)

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used *double deep Q-learning*, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close.

Did I miss something?

**Tags:** reinforcement-learning, ai-design, q-learning, deep-rl

> **Comments:**
> **username_1**: It does sound like something's off.  Have you already used aggregates of multiple frames so that the movement of the ball and paddle are captured?  My own solutions to this a few years back easily achieved perfect/infinite games after a few hours.  

---

## 1 answers

### Answer by username_2 on 2020-06-08. Score: 0

To my knowledge from reading about model-based and model-free reinforcement learning,

DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):

<https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1>

You should see under the heading "Definition" (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: "model-free techniques require extensive experience."

Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.

Also there are a lot of states: there are ($256^{128}$) states. That is a really big number (I'm just emphasizing that training may take a long while)

Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: <https://arxiv.org/pdf/1509.06461.pdf> . I am assuming they used image data as well because Double DQN was compared to DQN.


---

