Answers.Body: <p>Sergey Levine's comment does seem confusing, here's the clarification. The policy gradient derivation itself does not explicitly require the Markov property. Instead, it operates on the probability distribution of trajectories under the parameterized policy <span class="math-container">$\pi_{\theta}$</span>, which uses the Markov assumption since in most RL with fully observable states any trajectory is generated by a MDP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-28T06:59:44.820Z
Answers.LastActivityDate: 2024-11-28T06:59:44.820Z
Answers.UserDisplayName: cinch
Answers.Body: <p>I think the equation doesn't check out, when <span class="math-container">$i$</span> runs to <span class="math-container">$T$</span>, we don't have <span class="math-container">$s_{T+1}$</span> to plug in to <span class="math-container">$s_{i+1}$</span> in your first equation. Sorry it's not an answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-27T16:12:21.920Z
Answers.LastActivityDate: 2024-11-27T16:12:21.920Z
Answers.UserDisplayName: Son Do
Body: <p>To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro.html" rel="nofollow noreferrer">spinningup tutorial</a>):</p>
  <p><span class="math-container">$$
  \begin{align}
  P_\theta(\tau) &amp;= P_\theta(s_0, a_0, s_1, a_1, \dots, s_T, a_T) \\
  &amp; = p(s_0) \prod_{i=0}^T \pi_\theta(a_i | s_i) p(s_{i+1} | s_i, a_i)
  \end{align}
  $$</span></p>
  <p>The expression is based on the chain rule for probability. My understanding is that the application of the chain rule should give up this expression:</p>
  <p><span class="math-container">$$
  p(s_0)\prod_{i=0}^T \pi_\theta(a_i|s_i, a_{i-1}, s_{i-1}, a_{i-2}, \dots, s_0, a_0) p(s_{i+1} | s_i, a_i, s_{i-1}, a_{i-1}, \dots, a_0, s_0)
  $$</span></p>
  <p>Then the Markov property should be applicable, producing the desired equality. This should only depend on the latest state-action pair.</p>
  <p><strong>Here are my questions:</strong></p>
  <ol>
  <li><p>Is this true?</p>
  </li>
  <li><p>I watched this lecture about policy gradients, and <a href="https://youtu.be/VSPYKXm_hMA?list=PL_iWQOsE6TfURIIhCrlt-wj9ByIVpbfGc&amp;t=376" rel="nofollow noreferrer">at this time</a> during the lecture, Sergey says that: &quot;at no point did we use the Markov property when we derived the policy gradient&quot;, which left me confused. I assumed that the initial step of calculating the trajectory probability was using the Markov property.</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-21T00:27:28.590Z
Comments.Text: in the first equation you show, you definitely are using the Markov Property. When deriving the policy gradient you don’t _explicitly_ use the Markov property, if you refer to the Sutton and Barto derivation this is clear. However, I am still of the opinion that the Markov property is used, as the underlying assumption of the MDP is that the Markov property holds (eg our policy is only conditioned on the current state, not the whole trajectory)
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-21T00:58:12.527Z
Comments.Text: I watched the videos you’re watching and I would argue that the way he derives the policy gradient definitely does use the Markov property — he directly uses the probability of the trajectory that you have in your question and the LHS = RHS only if you assume the Markov property holds, otherwise as you say you would end up with a term like the second equation you have written.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-21T01:19:01.220Z
Comments.Text: I still have to read the policy gradient chapter in Sutton & Barto. Maybe comparing the two derivation can clear it up. I wonder if conditioning on policy is the answer, so it should be $P(\tau | \pi)$. I just realized thats how its written in the spinning up docs.
Comments.UserDisplayName: Gerges
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-21T01:23:09.867Z
Comments.Text: I don’t know what they mean by ‘conditioning on a policy’ — that is analogous to conditioning on a density function.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><policy-gradients><markov-property>
Title: Policy gradient: Does it use the Markov property?
UserDisplayName: Gerges
ThreadText: # Policy gradient: Does it use the Markov property?
  
  (Asked by: username_0 on 2020-12-20)
  
  To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see [spinningup tutorial](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)):
  
  $$
  \begin{align}
  P\_\theta(\tau) &= P\_\theta(s\_0, a\_0, s\_1, a\_1, \dots, s\_T, a\_T) \\
  & = p(s\_0) \prod\_{i=0}^T \pi\_\theta(a\_i | s\_i) p(s\_{i+1} | s\_i, a\_i)
  \end{align}
  $$
  
  The expression is based on the chain rule for probability. My understanding is that the application of the chain rule should give up this expression:
  
  $$
  p(s\_0)\prod\_{i=0}^T \pi\_\theta(a\_i|s\_i, a\_{i-1}, s\_{i-1}, a\_{i-2}, \dots, s\_0, a\_0) p(s\_{i+1} | s\_i, a\_i, s\_{i-1}, a\_{i-1}, \dots, a\_0, s\_0)
  $$
  
  Then the Markov property should be applicable, producing the desired equality. This should only depend on the latest state-action pair.
  
  **Here are my questions:**
  
  1. Is this true?
  2. I watched this lecture about policy gradients, and [at this time](https://youtu.be/VSPYKXm_hMA?list=PL_iWQOsE6TfURIIhCrlt-wj9ByIVpbfGc&t=376) during the lecture, Sergey says that: "at no point did we use the Markov property when we derived the policy gradient", which left me confused. I assumed that the initial step of calculating the trajectory probability was using the Markov property.
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, markov-property
  > **Comments:**
  > **username_1**: in the first equation you show, you definitely are using the Markov Property. When deriving the policy gradient you don’t _explicitly_ use the Markov property, if you refer to the Sutton and Barto derivation this is clear. However, I am still of the opinion that the Markov property is used, as the underlying assumption of the MDP is that the Markov property holds (eg our policy is only conditioned on the current state, not the whole trajectory)  
  > **username_1**: I watched the videos you’re watching and I would argue that the way he derives the policy gradient definitely does use the Markov property — he directly uses the probability of the trajectory that you have in your question and the LHS = RHS only if you assume the Markov property holds, otherwise as you say you would end up with a term like the second equation you have written.  
  > **username_0**: I still have to read the policy gradient chapter in Sutton & Barto. Maybe comparing the two derivation can clear it up. I wonder if conditioning on policy is the answer, so it should be $P(\tau | \pi)$. I just realized thats how its written in the spinning up docs.  
  > **username_1**: I don’t know what they mean by ‘conditioning on a policy’ — that is analogous to conditioning on a density function.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-11-28. Score: 0
  
  Sergey Levine's comment does seem confusing, here's the clarification. The policy gradient derivation itself does not explicitly require the Markov property. Instead, it operates on the probability distribution of trajectories under the parameterized policy $\pi\_{\theta}$, which uses the Markov assumption since in most RL with fully observable states any trajectory is generated by a MDP.
  
  
  ---
  
  ### Answer by username_3 on 2024-11-27. Score: 0
  
  I think the equation doesn't check out, when $i$ runs to $T$, we don't have $s\_{T+1}$ to plug in to $s\_{i+1}$ in your first equation. Sorry it's not an answer.
  
  
  ---
  
  