<p><strong>Equation 1</strong></p>
<p>In normal Q-Learning your target is defined as <span class="math-container">$y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$</span>. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations <strong>for each image augmentation</strong>. To turn this into the expected value over all <span class="math-container">$k$</span> transformations for the given state we need to average it by dividing the summed targets by the number of transformations <span class="math-container">${k}$</span>.</p>
<p><strong>Equation 3</strong></p>
<p>Here the Q-Function is updated with respect to all the image transformations. <span class="math-container">$f(s_i, v_{i,m})$</span> is the transformed image, i.e. it is the same as <span class="math-container">$s_i$</span> but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target <span class="math-container">$y_i$</span> averaged by the number of images transformations and states.</p>

CC BY-SA 4.0
2021-09-04T17:03:03.123Z
2021-09-06T17:51:21.120Z
2021-09-06T17:51:21.120Z
tnfru
<p>This <a href="https://arxiv.org/pdf/2004.13649.pdf" rel="nofollow noreferrer">paper</a> uses image augmentation to improve RL algorithms. It contains the following paragraph -
&quot;Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:</p>
<ol>
<li>transformations of the input image (Section 3.1).</li>
<li>averaging the Q target over K image transformations (Equation (1)).</li>
<li>averaging the Q function itself over M image transformations (Equation (3)).&quot;</li>
</ol>
<p>I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.</p>
<p>Here are the equations -</p>
<p><a href="https://i.sstatic.net/FoKeH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FoKeH.png" alt="enter image description here" /></a></p>
<p><a href="https://i.sstatic.net/YncwU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YncwU.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2021-07-26T22:15:50.930Z
If possible, you can update with equations mentioned in the paper along with notations for completeness.
hanugm
<reinforcement-learning><deep-learning><deep-rl><image-processing>
What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?
desert_ranger
# What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?

(Asked by: username_0 on 2021-07-26)

This [paper](https://arxiv.org/pdf/2004.13649.pdf) uses image augmentation to improve RL algorithms. It contains the following paragraph -
"Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:

1. transformations of the input image (Section 3.1).
2. averaging the Q target over K image transformations (Equation (1)).
3. averaging the Q function itself over M image transformations (Equation (3))."

I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.

Here are the equations -

[![enter image description here](https://i.sstatic.net/FoKeH.png)](https://i.sstatic.net/FoKeH.png)

[![enter image description here](https://i.sstatic.net/YncwU.png)](https://i.sstatic.net/YncwU.png)

**Tags:** reinforcement-learning, deep-learning, deep-rl, image-processing

> **Comments:**
> **username_1**: If possible, you can update with equations mentioned in the paper along with notations for completeness.  

---

## 1 answers

### Answer by username_2 on 2021-09-04. Score: 2 | Accepted answer

**Equation 1**

In normal Q-Learning your target is defined as $y\_t = r\_t + \gamma \mathrm{max\_a}Q(s\_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations **for each image augmentation**. To turn this into the expected value over all $k$ transformations for the given state we need to average it by dividing the summed targets by the number of transformations ${k}$.

**Equation 3**

Here the Q-Function is updated with respect to all the image transformations. $f(s\_i, v\_{i,m})$ is the transformed image, i.e. it is the same as $s\_i$ but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target $y\_i$ averaged by the number of images transformations and states.


---

