Answers.Body: <p>The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective.</p>
  
  <p>In <a href="https://arxiv.org/pdf/1502.05477v1" rel="nofollow noreferrer">the first version of the paper you linked</a>, they start out in Section 2 by defining Markov Decision Processes (MDPs) as tuples that, among other things, have a <strong>cost function</strong> $c : \mathcal{S} \rightarrow \mathbb{R}$. They define $\eta(\pi)$ as the expected discounted <strong>cost</strong> of a policy $\pi$, and subsequently also define state-action value functions $Q_{\pi}(s_t, a_t)$, value functions $V_{\pi}(s_t)$, and advantage functions $A_{\pi}(s, a)$ in terms of <strong>costs</strong>. Ultimately, in Equation 15, they write the following:</p>
  
  <p>\begin{aligned}
  \underset{\theta}{\text{minimize }} &amp; \mathbb{E}_{s \sim \rho_{\theta_{\text{old}}}, a \sim q} \left[ \frac{\pi_{\theta}(a \vert s)}{q(a \vert s)} Q_{\theta_{\text{old}}}(s, a) \right] \\ 
  \text{subject to } &amp; \mathbb{E}_{s \sim \rho_{\theta_{\text{old}}}} \left[ D_{KL}(\pi_{\theta_{\text{old}}}(\cdot \vert s) ~ \vert \vert ~ \pi_{\theta}(\cdot \vert s)) \right] \leq \delta
  \end{aligned}</p>
  
  <p>Now, there's a lot going on there, but we can <strong>very informally</strong> "simplify" it to only the parts that are relevant for this question, as follows:</p>
  
  <p>$$\underset{\theta}{\text{minimize }} \mathbb{E} \left[ Q(s, a) \right]$$</p>
  
  <p>When we look at just that, we see that we're pretty much trying to minimize $Q$-values, which are <strong>costs</strong>; that makes sense, typically <strong>costs</strong> are things we want to <strong>minimize</strong>.</p>
  
  <hr>
  
  <p>In the <a href="https://arxiv.org/pdf/1502.05477" rel="nofollow noreferrer">second version of the paper you linked</a>, they have changed the Preliminaries in Section 2. Now they no longer have a cost function $c$ in their definition of an MDP; they have replaced it by a <strong>reward function</strong> $r : \mathcal{S} \rightarrow \mathbb{R}$. Then they move on to define $\eta(\pi)$ as the <strong>expected discounted reward</strong> (rather than expected discounted cost), and also define $Q$, $V$ and $A$ in terms of <strong>rewards</strong> rather than costs. This now all matches the standard, common terminology in Reinforcement Learning.</p>
  
  <p>Ultimately, Equation 14 looks identical to what we saw above, it's again about an expectation of $Q$-values. But, now $Q$-values are <strong>rewards</strong> rather than <strong>costs</strong>. Rewards are generally things we want to <strong>maximize</strong>, rather than <strong>minimize</strong>, so that's why the objective swapped around.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-11T15:09:21.190Z
Answers.LastActivityDate: 2018-08-11T17:34:32.210Z
Answers.LastEditDate: 2018-08-11T17:34:32.210Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I happened to discover that the <a href="https://arxiv.org/pdf/1502.05477v1" rel="nofollow noreferrer">v1</a> (19 Feb 2015) and the <a href="https://arxiv.org/pdf/1502.05477" rel="nofollow noreferrer">v5</a> (20 Apr 2017) versions of TRPO papers have two different conclusions. The Equation (15) in v1 is <span class="math-container">$\min_\theta$</span> while the Equation (14) in v2 is <span class="math-container">$\max_\theta$</span>. So, I'm a little bit confused about which one to choose.</p>
  
  <p>BTW, I found that in the <a href="https://arxiv.org/pdf/1506.02438" rel="nofollow noreferrer">High-Dimensional Continuous Control Using Generalized Advantage Estimation</a>, the Equation (31) uses <span class="math-container">$\min_\theta$</span>.</p>
  
Tags: <reinforcement-learning><optimization><deep-rl><trust-region-policy-optimization>
Title: Maximizing or Minimizing in Trust Region Policy Optimization?
UserDisplayName: fish_tree
ThreadText: # Maximizing or Minimizing in Trust Region Policy Optimization?
  
  (Asked by: username_0 on 2018-07-15)
  
  I happened to discover that the [v1](https://arxiv.org/pdf/1502.05477v1) (19 Feb 2015) and the [v5](https://arxiv.org/pdf/1502.05477) (20 Apr 2017) versions of TRPO papers have two different conclusions. The Equation (15) in v1 is $\min\_\theta$ while the Equation (14) in v2 is $\max\_\theta$. So, I'm a little bit confused about which one to choose.
  
  BTW, I found that in the [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://arxiv.org/pdf/1506.02438), the Equation (31) uses $\min\_\theta$.
  
  **Tags:** reinforcement-learning, optimization, deep-rl, trust-region-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-11. Score: 3 | Accepted answer
  
  The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective.
  
  In [the first version of the paper you linked](https://arxiv.org/pdf/1502.05477v1), they start out in Section 2 by defining Markov Decision Processes (MDPs) as tuples that, among other things, have a **cost function** $c : \mathcal{S} \rightarrow \mathbb{R}$. They define $\eta(\pi)$ as the expected discounted **cost** of a policy $\pi$, and subsequently also define state-action value functions $Q\_{\pi}(s\_t, a\_t)$, value functions $V\_{\pi}(s\_t)$, and advantage functions $A\_{\pi}(s, a)$ in terms of **costs**. Ultimately, in Equation 15, they write the following:
  
  \begin{aligned}
  \underset{\theta}{\text{minimize }} & \mathbb{E}\_{s \sim \rho\_{\theta\_{\text{old}}}, a \sim q} \left[ \frac{\pi\_{\theta}(a \vert s)}{q(a \vert s)} Q\_{\theta\_{\text{old}}}(s, a) \right] \\
  \text{subject to } & \mathbb{E}\_{s \sim \rho\_{\theta\_{\text{old}}}} \left[ D\_{KL}(\pi\_{\theta\_{\text{old}}}(\cdot \vert s) ~ \vert \vert ~ \pi\_{\theta}(\cdot \vert s)) \right] \leq \delta
  \end{aligned}
  
  Now, there's a lot going on there, but we can **very informally** "simplify" it to only the parts that are relevant for this question, as follows:
  
  $$\underset{\theta}{\text{minimize }} \mathbb{E} \left[ Q(s, a) \right]$$
  
  When we look at just that, we see that we're pretty much trying to minimize $Q$-values, which are **costs**; that makes sense, typically **costs** are things we want to **minimize**.
  
  ---
  
  In the [second version of the paper you linked](https://arxiv.org/pdf/1502.05477), they have changed the Preliminaries in Section 2. Now they no longer have a cost function $c$ in their definition of an MDP; they have replaced it by a **reward function** $r : \mathcal{S} \rightarrow \mathbb{R}$. Then they move on to define $\eta(\pi)$ as the **expected discounted reward** (rather than expected discounted cost), and also define $Q$, $V$ and $A$ in terms of **rewards** rather than costs. This now all matches the standard, common terminology in Reinforcement Learning.
  
  Ultimately, Equation 14 looks identical to what we saw above, it's again about an expectation of $Q$-values. But, now $Q$-values are **rewards** rather than **costs**. Rewards are generally things we want to **maximize**, rather than **minimize**, so that's why the objective swapped around.
  
  
  ---
  
  