Answers.Body: <p><strong>Equation 1</strong></p>
  <p>In normal Q-Learning your target is defined as <span class="math-container">$y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$</span>. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations <strong>for each image augmentation</strong>. To turn this into the expected value over all <span class="math-container">$k$</span> transformations for the given state we need to average it by dividing the summed targets by the number of transformations <span class="math-container">${k}$</span>.</p>
  <p><strong>Equation 3</strong></p>
  <p>Here the Q-Function is updated with respect to all the image transformations. <span class="math-container">$f(s_i, v_{i,m})$</span> is the transformed image, i.e. it is the same as <span class="math-container">$s_i$</span> but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target <span class="math-container">$y_i$</span> averaged by the number of images transformations and states.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T17:03:03.123Z
Answers.LastActivityDate: 2021-09-06T17:51:21.120Z
Answers.LastEditDate: 2021-09-06T17:51:21.120Z
Answers.UserDisplayName: tnfru
Body: <p>This <a href="https://arxiv.org/pdf/2004.13649.pdf" rel="nofollow noreferrer">paper</a> uses image augmentation to improve RL algorithms. It contains the following paragraph -
  &quot;Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:</p>
  <ol>
  <li>transformations of the input image (Section 3.1).</li>
  <li>averaging the Q target over K image transformations (Equation (1)).</li>
  <li>averaging the Q function itself over M image transformations (Equation (3)).&quot;</li>
  </ol>
  <p>I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.</p>
  <p>Here are the equations -</p>
  <p><a href="https://i.sstatic.net/FoKeH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FoKeH.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/YncwU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YncwU.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-26T22:15:50.930Z
Comments.Text: If possible, you can update with equations mentioned in the paper along with notations for completeness.
Comments.UserDisplayName: hanugm
Tags: <reinforcement-learning><deep-learning><deep-rl><image-processing>
Title: What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?
UserDisplayName: desert_ranger
ThreadText: # What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?
  
  (Asked by: username_0 on 2021-07-26)
  
  This [paper](https://arxiv.org/pdf/2004.13649.pdf) uses image augmentation to improve RL algorithms. It contains the following paragraph -
  "Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:
  
  1. transformations of the input image (Section 3.1).
  2. averaging the Q target over K image transformations (Equation (1)).
  3. averaging the Q function itself over M image transformations (Equation (3))."
  
  I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.
  
  Here are the equations -
  
  [![enter image description here](https://i.sstatic.net/FoKeH.png)](https://i.sstatic.net/FoKeH.png)
  
  [![enter image description here](https://i.sstatic.net/YncwU.png)](https://i.sstatic.net/YncwU.png)
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl, image-processing
  > **Comments:**
  > **username_1**: If possible, you can update with equations mentioned in the paper along with notations for completeness.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-09-04. Score: 2 | Accepted answer
  
  **Equation 1**
  
  In normal Q-Learning your target is defined as $y\_t = r\_t + \gamma \mathrm{max\_a}Q(s\_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations **for each image augmentation**. To turn this into the expected value over all $k$ transformations for the given state we need to average it by dividing the summed targets by the number of transformations ${k}$.
  
  **Equation 3**
  
  Here the Q-Function is updated with respect to all the image transformations. $f(s\_i, v\_{i,m})$ is the transformed image, i.e. it is the same as $s\_i$ but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target $y\_i$ averaged by the number of images transformations and states.
  
  
  ---
  
  