Answers.Body: <p><strong>TL;DR</strong> In the DQN paper, each environment was trained for 50 million frames, grouped in fours without overlap, so there were 12.5 million state, action, reward next-state records used.</p>
  <p>The above direct numerical answer to the question as posed is not the whole story though.</p>
  <blockquote>
  <p>I'm trying to understand the dataset and how many states, actions and rewards have been used to train the agent on a single game.</p>
  </blockquote>
  <p>It seems you are viewing Reinforcement Learning (RL) through terms you understand from supervised learning. Although there are some similarities, especially when considering the neural network that performs function approximation inside most deep RL, there are important differences which mean the information you are looking for doesn't really exist.</p>
  <p>The main purpose of RL is for an agent to learn through experience that it also helps to generate, through trial and error. Although off-policy algorithms such as DQN could learn from fixed datasets (and there are some circumstances where they might, perhaps as part of a more complex pipeline), this is not what the papers you reference are doing.</p>
  <p>RL experiments can be repeated provided all random number generators (in the learning agent and in the environment) are seeded the same way. That may be of interest to researchers wanting to replicate work exactly, but more generally useful are statistical measures of &quot;typical&quot; learning graphs. The environment defines the problem to be solved, and the exact data generated through the RL agent exploring it in a set of runs is not necessarily as interesting when interpreting the results, or considering whether to use a certain type of agent for a specific problem. <em>Having a reference copy of the environment identical to the one used in the paper is the equivalent in RL of having the &quot;dataset&quot;.</em></p>
  <p>You could log the state, action, rewards data from each experiment (and in DQN it is already being stored temporarily in the experience replay table), but you would not expect it to repeat on the next run even with the same agent. Also, when comparing to some other agent (or agent hyperparameters), one major point of difference is that you would <em>expect</em> that agent to generate a different set of states, actions and rewards - choosing how and when to explore different parts of the environment is a key differentator between agents. So it is usually not considered a critical part of the work to log every state. That's not to say that researchers won't do it - presumably many do keep records for debugging or simply to be thorough. However, more important in published RL results is to characterise the learning curves against some metric such as the amount of experience required, or another cost such as total CPU resource or time used in training.</p>
  <p>In the original DQN paper, I believe that each 4 frames were stacked separately into a single state, without overlap, thus each 4 frames equals one state, for which a single action would be taken and then reward and next state observed. So dividing the frame count by 4 should give you the &quot;dataset size&quot;. But it is important to note that this was never used as a whole in a supervised learning manner, and was not re-used between runs. The experience replay buffer was sampled randomly on each timestep, and there was an average number of times each sample was used (32 I think) to train the network - you might compare that to running 32 epochs of training across all state/action pairs as inputs, but it is only a loose comparison due to the sample-with-replacement sampling strategy and the sliding window effect of the experience replay.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-14T12:52:34.630Z
Answers.Text: Thanks for clarifying the differences and other stuffs, but I'm looking to explore a method that uses less interaction between the environment and agent, so I think the number of trials should be equal to the number of actions made by the agent, also equal to the number of rewards. In this case I can say that the performance measures of number of trials is better or worse. That's why I need to know the number of actions made by an agent in a single game.
Answers.UserDisplayName: Jigberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-14T12:58:17.060Z
Answers.Text: @Jigberto: Your question is how many states/actions and rewards for a single game. I have now answered that at the top (it is 12.5 million per game). Unless you mean by "per game" per episode (i.e. number of frames from starting a game session to completing it due to winning or losing all lives or some kind of timeout/reset). The issue with per episode is that it will vary hugely per type of game and how well the agent is currently performing. To the point that is a meaningless statistic in the papers you reference, except for games that reward simple survival time.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-14T13:04:16.027Z
Answers.Text: In those articles were mentioned different number of frames like: 1M, 10M, 50M, 200M. In the second article was mentioned "... Training is done over 50M steps (i.e., 200M frames)." which seems like you said 200M / 4 = 50M actions(steps) ... still not sure
Answers.UserDisplayName: Jigberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-14T13:09:55.747Z
Answers.Text: I took 50M frames from the text of DQN paper and have not read the DDQN paper. It is entirely possible that learning curves or results are published from multiple points in the training proces for comparison. This would be useful data to understand how the agent performs. For the original DQN paper I think you can safely divide any frame count given by 4 to get number of steps, which will equal the number of states, actions and rewards.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-14T12:23:15.610Z
Answers.LastActivityDate: 2022-05-14T12:51:31.750Z
Answers.LastEditDate: 2022-05-14T12:51:31.750Z
Answers.UserDisplayName: Neil Slater
Body: <p>Reading this two articles about Reinforcement Learning:</p>
  <ol>
  <li><a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">Deep Reinforcement Learning with Double Q-learning</a> by Hado van Hasselt et al.</li>
  <li><a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">Human-level control through deep reinforcement learning</a> by Volodymyr Mnih et al.</li>
  </ol>
  <p>I'm trying to understand the dataset and how many states, actions and rewards have been used to train the agent on a single game. I presume they should be equal, if an action is made on some state that generates some reward.</p>
  <p>The first article compares their results with second's results. Both refers to different count of frames, however it is not clear how many actions, rewards and states have the dataset of a single game. More than that, each game could have random states, actions, rewards, which is somehow an imprecise comparison, but I could not find a dataset used for first article and second one.</p>
  <p>Please, somebody who have a better understanding tell what is the total number of states, actions and rewards for a single game that the agent was trained on, because I'm trying to understand the dataset, and if there exists such dataset please show the source.</p>
  
Tags: <reinforcement-learning><deep-learning><papers><dqn><datasets>
Title: What is the total number of actions and rewards count
UserDisplayName: Jigberto
ThreadText: # What is the total number of actions and rewards count
  
  (Asked by: username_0 on 2022-05-13)
  
  Reading this two articles about Reinforcement Learning:
  
  1. [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/pdf/1509.06461.pdf) by Hado van Hasselt et al.
  2. [Human-level control through deep reinforcement learning](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf) by Volodymyr Mnih et al.
  
  I'm trying to understand the dataset and how many states, actions and rewards have been used to train the agent on a single game. I presume they should be equal, if an action is made on some state that generates some reward.
  
  The first article compares their results with second's results. Both refers to different count of frames, however it is not clear how many actions, rewards and states have the dataset of a single game. More than that, each game could have random states, actions, rewards, which is somehow an imprecise comparison, but I could not find a dataset used for first article and second one.
  
  Please, somebody who have a better understanding tell what is the total number of states, actions and rewards for a single game that the agent was trained on, because I'm trying to understand the dataset, and if there exists such dataset please show the source.
  
  **Tags:** reinforcement-learning, deep-learning, papers, dqn, datasets
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-14. Score: 3 | Accepted answer
  
  **TL;DR** In the DQN paper, each environment was trained for 50 million frames, grouped in fours without overlap, so there were 12.5 million state, action, reward next-state records used.
  
  The above direct numerical answer to the question as posed is not the whole story though.
  
  > I'm trying to understand the dataset and how many states, actions and rewards have been used to train the agent on a single game.
  
  It seems you are viewing Reinforcement Learning (RL) through terms you understand from supervised learning. Although there are some similarities, especially when considering the neural network that performs function approximation inside most deep RL, there are important differences which mean the information you are looking for doesn't really exist.
  
  The main purpose of RL is for an agent to learn through experience that it also helps to generate, through trial and error. Although off-policy algorithms such as DQN could learn from fixed datasets (and there are some circumstances where they might, perhaps as part of a more complex pipeline), this is not what the papers you reference are doing.
  
  RL experiments can be repeated provided all random number generators (in the learning agent and in the environment) are seeded the same way. That may be of interest to researchers wanting to replicate work exactly, but more generally useful are statistical measures of "typical" learning graphs. The environment defines the problem to be solved, and the exact data generated through the RL agent exploring it in a set of runs is not necessarily as interesting when interpreting the results, or considering whether to use a certain type of agent for a specific problem. *Having a reference copy of the environment identical to the one used in the paper is the equivalent in RL of having the "dataset".*
  
  You could log the state, action, rewards data from each experiment (and in DQN it is already being stored temporarily in the experience replay table), but you would not expect it to repeat on the next run even with the same agent. Also, when comparing to some other agent (or agent hyperparameters), one major point of difference is that you would *expect* that agent to generate a different set of states, actions and rewards - choosing how and when to explore different parts of the environment is a key differentator between agents. So it is usually not considered a critical part of the work to log every state. That's not to say that researchers won't do it - presumably many do keep records for debugging or simply to be thorough. However, more important in published RL results is to characterise the learning curves against some metric such as the amount of experience required, or another cost such as total CPU resource or time used in training.
  
  In the original DQN paper, I believe that each 4 frames were stacked separately into a single state, without overlap, thus each 4 frames equals one state, for which a single action would be taken and then reward and next state observed. So dividing the frame count by 4 should give you the "dataset size". But it is important to note that this was never used as a whole in a supervised learning manner, and was not re-used between runs. The experience replay buffer was sampled randomly on each timestep, and there was an average number of times each sample was used (32 I think) to train the network - you might compare that to running 32 epochs of training across all state/action pairs as inputs, but it is only a loose comparison due to the sample-with-replacement sampling strategy and the sliding window effect of the experience replay.
  
  > **Comments:**
  > **username_0**: Thanks for clarifying the differences and other stuffs, but I'm looking to explore a method that uses less interaction between the environment and agent, so I think the number of trials should be equal to the number of actions made by the agent, also equal to the number of rewards. In this case I can say that the performance measures of number of trials is better or worse. That's why I need to know the number of actions made by an agent in a single game.  
  > **username_1**: @username_0: Your question is how many states/actions and rewards for a single game. I have now answered that at the top (it is 12.5 million per game). Unless you mean by "per game" per episode (i.e. number of frames from starting a game session to completing it due to winning or losing all lives or some kind of timeout/reset). The issue with per episode is that it will vary hugely per type of game and how well the agent is currently performing. To the point that is a meaningless statistic in the papers you reference, except for games that reward simple survival time.  
  > **username_0**: In those articles were mentioned different number of frames like: 1M, 10M, 50M, 200M. In the second article was mentioned "... Training is done over 50M steps (i.e., 200M frames)." which seems like you said 200M / 4 = 50M actions(steps) ... still not sure  
  > **username_1**: I took 50M frames from the text of DQN paper and have not read the DDQN paper. It is entirely possible that learning curves or results are published from multiple points in the training proces for comparison. This would be useful data to understand how the agent performs. For the original DQN paper I think you can safely divide any frame count given by 4 to get number of steps, which will equal the number of states, actions and rewards.  
  
  ---
  
  