Answers.Body: <p>The goals of experience replay as first proposed by <a href="https://link.springer.com/content/pdf/10.1007/BF00992699.pdf" rel="nofollow noreferrer">Lin (1992)</a> and more recently applied successfully in the DQN algorithm by <a href="https://arxiv.org/pdf/1312.5602.pdf" rel="nofollow noreferrer">Mnih et al. (2013)</a> are to break temporal correlations of updates and to prevent forgetting of experiences that might be useful later on.</p>
  <p>To meet these goals, the replay buffer should store tuples required in the learning step.</p>
  <p>Most works that use experience replay, including those mentioned before, learn (to approximate) the Q function, i.e. <span class="math-container">$Q(s,a)$</span>. Clearly, this function depends on the sampled action <span class="math-container">$a$</span>.</p>
  <p>If <span class="math-container">$a$</span> is not used at all in the training process then it would not need to be stored in the replay buffer. In such a scenario, however, the problems that motivate the replay buffer may not be present in the first place.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T13:06:56.560Z
Answers.LastActivityDate: 2021-02-22T13:06:56.560Z
Answers.UserDisplayName: Floris den Hengst
Answers.Body: <p>We need to store the action <span class="math-container">$a$</span> as it tells us the action that we took in the state that we are backing up.</p>
  <p>Suppose we are in state <span class="math-container">$s$</span> and we take action <span class="math-container">$a$</span>, then we will receive a reward <span class="math-container">$r$</span> and next state <span class="math-container">$s'$</span>. The goal of RL, and in particular DQN (I mention DQN as it is the first algorithm that comes to mind when I think of a replay buffer but it is of course not the only algorithm to make use of one), is that we are trying to learn optimal state-action value functions <span class="math-container">$Q(s, a)$</span>. We thus want our value function to be able to predict <span class="math-container">$y = r + \gamma \max_{a'}Q(s', a')$</span>, i.e. given <span class="math-container">$s$</span> and <span class="math-container">$a$</span> we want to be able to predict <span class="math-container">$y$</span>. As you can see, we need to know which action we took in state <span class="math-container">$s$</span> so that we can train our value function to approximate <span class="math-container">$y$</span>, and the value function clearly depends on <span class="math-container">$a$</span>, hence we need to also store <span class="math-container">$a$</span> in our experience tuple.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-23T12:14:33.060Z
Answers.Text: Perhaps I now understand where my mistake was. Do I understand correctly, that for calculating the loss and performing gradient descent, we do not actually compare all of the Q-values, but only the one that corresponds to $Q(s,a)$ to its target value? Until now, I was under the impression we would do something like mean-squared-error, where we compare all the Q-values to their "labels" or rather the targets we calculated using $s'$ and $r$.
Answers.UserDisplayName: bertushunzius
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-23T14:36:52.163Z
Answers.Text: You're correct that we only update the Q-value for one action. I posted an answered [here](https://ai.stackexchange.com/a/26163/36821) that seems like it might explain this in more detail.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-23T14:51:37.690Z
Answers.Text: I see. I think that answers my question, then :-) Thank you very much
Answers.UserDisplayName: bertushunzius
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-22T14:58:29.630Z
Answers.LastActivityDate: 2021-02-22T14:58:29.630Z
Answers.UserDisplayName: David
Body: <p>From what I understand, experience replay works by storing tuples of <span class="math-container">$(s, a, r, s')$</span>  to be sampled for training. I understand why we store <span class="math-container">$s$</span>, <span class="math-container">$r$</span> and <span class="math-container">$s'$</span>. However, I do not understand the need for storing the action <span class="math-container">$a$</span>.</p>
  <p>As I recall, the reward <span class="math-container">$r$</span> and the next state <span class="math-container">$s'$</span> are both used to calculate the target values. We can then compare these target values to the output we get when we do a forward-pass using state <span class="math-container">$s$</span> It seems to me that the stored action <span class="math-container">$a$</span> is not required for this process to work; or am I missing something? Why would we store the action <span class="math-container">$a$</span> if it isn't used in the training process itself?</p>
  <p>Please, forgive me if this question has been answered before. I looked, but was unable to find anything other than generic explanations as to what experience replay is and why we do it.</p>
  
Tags: <reinforcement-learning><experience-replay>
Title: What is the purpose of storing the action $a$ within an experience tuple?
UserDisplayName: bertushunzius
ThreadText: # What is the purpose of storing the action $a$ within an experience tuple?
  
  (Asked by: username_0 on 2021-02-22)
  
  From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for storing the action $a$.
  
  As I recall, the reward $r$ and the next state $s'$ are both used to calculate the target values. We can then compare these target values to the output we get when we do a forward-pass using state $s$ It seems to me that the stored action $a$ is not required for this process to work; or am I missing something? Why would we store the action $a$ if it isn't used in the training process itself?
  
  Please, forgive me if this question has been answered before. I looked, but was unable to find anything other than generic explanations as to what experience replay is and why we do it.
  
  **Tags:** reinforcement-learning, experience-replay
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-02-22. Score: 3 | Accepted answer
  
  We need to store the action $a$ as it tells us the action that we took in the state that we are backing up.
  
  Suppose we are in state $s$ and we take action $a$, then we will receive a reward $r$ and next state $s'$. The goal of RL, and in particular DQN (I mention DQN as it is the first algorithm that comes to mind when I think of a replay buffer but it is of course not the only algorithm to make use of one), is that we are trying to learn optimal state-action value functions $Q(s, a)$. We thus want our value function to be able to predict $y = r + \gamma \max\_{a'}Q(s', a')$, i.e. given $s$ and $a$ we want to be able to predict $y$. As you can see, we need to know which action we took in state $s$ so that we can train our value function to approximate $y$, and the value function clearly depends on $a$, hence we need to also store $a$ in our experience tuple.
  
  > **Comments:**
  > **username_0**: Perhaps I now understand where my mistake was. Do I understand correctly, that for calculating the loss and performing gradient descent, we do not actually compare all of the Q-values, but only the one that corresponds to $Q(s,a)$ to its target value? Until now, I was under the impression we would do something like mean-squared-error, where we compare all the Q-values to their "labels" or rather the targets we calculated using $s'$ and $r$.  
  > **username_2**: You're correct that we only update the Q-value for one action. I posted an answered [here](https://ai.stackexchange.com/a/26163/36821) that seems like it might explain this in more detail.  
  > **username_0**: I see. I think that answers my question, then :-) Thank you very much  
  
  ---
  
  ### Answer by username_1 on 2021-02-22. Score: 2
  
  The goals of experience replay as first proposed by [Lin (1992)](https://link.springer.com/content/pdf/10.1007/BF00992699.pdf) and more recently applied successfully in the DQN algorithm by [Mnih et al. (2013)](https://arxiv.org/pdf/1312.5602.pdf) are to break temporal correlations of updates and to prevent forgetting of experiences that might be useful later on.
  
  To meet these goals, the replay buffer should store tuples required in the learning step.
  
  Most works that use experience replay, including those mentioned before, learn (to approximate) the Q function, i.e. $Q(s,a)$. Clearly, this function depends on the sampled action $a$.
  
  If $a$ is not used at all in the training process then it would not need to be stored in the replay buffer. In such a scenario, however, the problems that motivate the replay buffer may not be present in the first place.
  
  
  ---
  
  