Answers.Body: <p>Even for a deterministic policy <span class="math-container">${\mu_\theta}$</span>, its induced trajectory <span class="math-container">$\tau$</span> in RL still follows a distribution influenced under the said policy <span class="math-container">${\mu_\theta}$</span> and there's usually a corresponding unique stationary distribution of all states (<span class="math-container">$s \sim \rho^{\mu_\theta}$</span>), assuming ergodicity from the <a href="https://en.wikipedia.org/wiki/Stochastic_matrix" rel="nofollow noreferrer">stochastic matrix</a> nature of the environment's Markov chain transition dynamics. Therefore from the common formulation of policy gradient method's objective function <span class="math-container">\begin{equation}
      J(\theta) = \underset{\tau \sim \mu_\theta}{\mathbb{E}}\left[R(\tau)\right] = \underset{s \sim \rho^{\mu_\theta}}{\mathbb{E}}\left[Q^{\mu_{\theta}}(s,\,a)\right] = \underset{s \sim \rho^{\mu_\theta}}{\mathbb{E}}\left[Q^{\mu_{\theta}}(s,\,\mu_\theta(s))\right]
  \end{equation}</span></p>
  <p>by the definition of the action value function at any state <span class="math-container">$s$</span> under a given policy which links to the cumulative rewards of a trajectory starting from <span class="math-container">$s$</span> as the reward-to-go. And then you take gradient along with chain rule, you can arrive at your last DPG's gradient formula above correctly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-01T18:58:04.243Z
Answers.Text: Yes, I initially assumed that it was due to the definition of the action value function and it's relationship with the expected return depending on both state and action taken. However, I was looking if there was some kind of demonstration with reasoned steps and transformations needed which would end up being simplified to the action value function, like for the stochastic policy, where the probability distribution over a trajectory.
Answers.UserDisplayName: marc_spector
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-02T03:21:41.810Z
Answers.Text: See long [derivation](https://stats.stackexchange.com/questions/512668/deriving-state-action-marginal-in-reinforcement-learning/657600). Yet my answer here is also rigorous, if you understand why we can substitute  the original trajectory random variable $\tau$ with the start state random variable $s$, which is nothing but what I've highlighted in my answer about the existence of a unique limiting stationary distribution per *ergodicity* of the induced Markov chain. See a recent [post](https://ai.stackexchange.com/questions/47263/conditions-for-the-existence-of-stationary-state-distribution).
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T22:36:54.507Z
Answers.LastActivityDate: 2024-12-01T06:54:24.983Z
Answers.LastEditDate: 2024-12-01T06:54:24.983Z
Answers.UserDisplayName: cinch
Body: <p>I have a question regarding how the expected return of a deterministic policy in written. I have seen that in some cases the use the Q-Function as it is shown in the part <em><strong>Objective function Deterministic Policy</strong></em>. However, I do not fully understand the steps how it is obtained contrary to the stochastic policy. What are the steps or reasoning to get the expression <em><strong>Objective function Deterministic Policy</strong></em> ?</p>
  <p><strong>Probability distribution of a trajectory <span class="math-container">$\tau$</span> given policy <span class="math-container">$\pi_\theta$</span></strong>
  <span class="math-container">\begin{equation}
      P(\tau\,|\,\pi_\theta) = \rho_0(s_0) \prod_{t=0}^{T-1}P(s_{t+1}\,|\,s_t,\,a_t)\pi_\theta(a_t\,|\,s_t)
  \end{equation}</span></p>
  <p><strong>Objective function (Stochastic Policy)</strong>
  <span class="math-container">\begin{equation}
      J(\theta) = \int_\tau P(\tau\,|\,\pi_\theta)R(\tau) = \underset{\tau \sim \pi_\theta}{\mathbb{E}}\left[R(\tau)\right]
  \end{equation}</span>
  Where:<br />
  <span class="math-container">$R(\tau)$</span> is the cumulative reward of the trajectory, being <span class="math-container">$$R(\tau) = \sum_{t=0}^{T}r_t$$</span> if
  finite-horizon undiscounted reward or <span class="math-container">$$R(\tau) = \sum_{t=0}^{\infty}\gamma^{\,t}r_t$$</span> if
  infinite-horizon discounted reward.</p>
  <p><strong>Probability distribution of a trajectory <span class="math-container">$\tau$</span> given deterministic policy <span class="math-container">$\mu_\theta$</span></strong>
  <span class="math-container">\begin{equation}
      P(\tau\,|\,\mu_\theta) = \rho_0(s_0) \prod_{t=0}^{T-1}P(s_{t+1}\,|\,s_t,\,\mu_\theta(s_t))
  \end{equation}</span>
  Given:<br />
  <span class="math-container">\begin{equation}
      \mu_\theta(a_t \mid s_t) = 
      \begin{cases} 
      1 &amp; \text{if } a_t = \mu_\theta(s_t) \\
      0 &amp; \text{if } a_t \neq \mu_\theta(s_t)
      \end{cases}
  \end{equation}</span></p>
  <p><strong>Objective function Deterministic Policy</strong>
  <span class="math-container">\begin{equation}
      J(\theta) = \underset{s \sim \rho^{\mu_\theta}}{\mathbb{E}}\left[Q^{\mu}(s,\,\mu_\theta(s))\right]
  \end{equation}</span></p>
  <p><strong>Stochastic Policy Gradient</strong>
  <span class="math-container">\begin{equation}
      \nabla_\theta J(\theta) = \underset{\tau \sim \pi_\theta}{\mathbb{E}}\left[\nabla_\theta\log\pi_\theta(\tau)R(\tau)\right]
  \end{equation}</span>
  <span class="math-container">\begin{equation}
      \nabla_\theta J(\theta) = \underset{\tau \sim \pi_\theta}{\mathbb{E}}\left[\nabla_\theta\log\pi_\theta(a\,|\,s)R(\tau)\right]
  \end{equation}</span></p>
  <p><strong>Deterministic Policy Gradient</strong>
  <span class="math-container">\begin{equation}
      \nabla_\theta J(\theta) = \underset{s \sim \rho^{\mu_\theta}}{\mathbb{E}}\left[\nabla_\theta \mu_\theta(s) \nabla_a Q^{\mu}(s,\,a)\,|_{a = \mu_\theta(s)}\right]
  \end{equation}</span></p>
  
Tags: <objective-functions><policy-gradients><policies><return><deterministic-policy>
Title: Expected return formula for deterministic policy
UserDisplayName: marc_spector
ThreadText: # Expected return formula for deterministic policy
  
  (Asked by: username_0 on 2024-11-30)
  
  I have a question regarding how the expected return of a deterministic policy in written. I have seen that in some cases the use the Q-Function as it is shown in the part ***Objective function Deterministic Policy***. However, I do not fully understand the steps how it is obtained contrary to the stochastic policy. What are the steps or reasoning to get the expression ***Objective function Deterministic Policy*** ?
  
  **Probability distribution of a trajectory $\tau$ given policy $\pi\_\theta$**
  \begin{equation}
  P(\tau\,|\,\pi\_\theta) = \rho\_0(s\_0) \prod\_{t=0}^{T-1}P(s\_{t+1}\,|\,s\_t,\,a\_t)\pi\_\theta(a\_t\,|\,s\_t)
  \end{equation}
  
  **Objective function (Stochastic Policy)**
  \begin{equation}
  J(\theta) = \int\_\tau P(\tau\,|\,\pi\_\theta)R(\tau) = \underset{\tau \sim \pi\_\theta}{\mathbb{E}}\left[R(\tau)\right]
  \end{equation}
  Where:  
  $R(\tau)$ is the cumulative reward of the trajectory, being $$R(\tau) = \sum\_{t=0}^{T}r\_t$$ if
  finite-horizon undiscounted reward or $$R(\tau) = \sum\_{t=0}^{\infty}\gamma^{\,t}r\_t$$ if
  infinite-horizon discounted reward.
  
  **Probability distribution of a trajectory $\tau$ given deterministic policy $\mu\_\theta$**
  \begin{equation}
  P(\tau\,|\,\mu\_\theta) = \rho\_0(s\_0) \prod\_{t=0}^{T-1}P(s\_{t+1}\,|\,s\_t,\,\mu\_\theta(s\_t))
  \end{equation}
  Given:  
  \begin{equation}
  \mu\_\theta(a\_t \mid s\_t) =
  \begin{cases}
  1 & \text{if } a\_t = \mu\_\theta(s\_t) \\
  0 & \text{if } a\_t \neq \mu\_\theta(s\_t)
  \end{cases}
  \end{equation}
  
  **Objective function Deterministic Policy**
  \begin{equation}
  J(\theta) = \underset{s \sim \rho^{\mu\_\theta}}{\mathbb{E}}\left[Q^{\mu}(s,\,\mu\_\theta(s))\right]
  \end{equation}
  
  **Stochastic Policy Gradient**
  \begin{equation}
  \nabla\_\theta J(\theta) = \underset{\tau \sim \pi\_\theta}{\mathbb{E}}\left[\nabla\_\theta\log\pi\_\theta(\tau)R(\tau)\right]
  \end{equation}
  \begin{equation}
  \nabla\_\theta J(\theta) = \underset{\tau \sim \pi\_\theta}{\mathbb{E}}\left[\nabla\_\theta\log\pi\_\theta(a\,|\,s)R(\tau)\right]
  \end{equation}
  
  **Deterministic Policy Gradient**
  \begin{equation}
  \nabla\_\theta J(\theta) = \underset{s \sim \rho^{\mu\_\theta}}{\mathbb{E}}\left[\nabla\_\theta \mu\_\theta(s) \nabla\_a Q^{\mu}(s,\,a)\,|\_{a = \mu\_\theta(s)}\right]
  \end{equation}
  
  **Tags:** objective-functions, policy-gradients, policies, return, deterministic-policy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-30. Score: 1 | Accepted answer
  
  Even for a deterministic policy ${\mu\_\theta}$, its induced trajectory $\tau$ in RL still follows a distribution influenced under the said policy ${\mu\_\theta}$ and there's usually a corresponding unique stationary distribution of all states ($s \sim \rho^{\mu\_\theta}$), assuming ergodicity from the [stochastic matrix](https://en.wikipedia.org/wiki/Stochastic_matrix) nature of the environment's Markov chain transition dynamics. Therefore from the common formulation of policy gradient method's objective function \begin{equation}
  J(\theta) = \underset{\tau \sim \mu\_\theta}{\mathbb{E}}\left[R(\tau)\right] = \underset{s \sim \rho^{\mu\_\theta}}{\mathbb{E}}\left[Q^{\mu\_{\theta}}(s,\,a)\right] = \underset{s \sim \rho^{\mu\_\theta}}{\mathbb{E}}\left[Q^{\mu\_{\theta}}(s,\,\mu\_\theta(s))\right]
  \end{equation}
  
  by the definition of the action value function at any state $s$ under a given policy which links to the cumulative rewards of a trajectory starting from $s$ as the reward-to-go. And then you take gradient along with chain rule, you can arrive at your last DPG's gradient formula above correctly.
  
  > **Comments:**
  > **username_0**: Yes, I initially assumed that it was due to the definition of the action value function and it's relationship with the expected return depending on both state and action taken. However, I was looking if there was some kind of demonstration with reasoned steps and transformations needed which would end up being simplified to the action value function, like for the stochastic policy, where the probability distribution over a trajectory.  
  > **username_1**: See long [derivation](https://stats.stackexchange.com/questions/512668/deriving-state-action-marginal-in-reinforcement-learning/657600). Yet my answer here is also rigorous, if you understand why we can substitute  the original trajectory random variable $\tau$ with the start state random variable $s$, which is nothing but what I've highlighted in my answer about the existence of a unique limiting stationary distribution per *ergodicity* of the induced Markov chain. See a recent [post](https://ai.stackexchange.com/questions/47263/conditions-for-the-existence-of-stationary-state-distribution).  
  
  ---
  
  