<p>I'm studying reinforcement learning now and I'm quite a newbie to this field.
I have some questions about how to derive the equation as below.</p>
<p><span class="math-container">$p_{\theta}(s_{1},a_{1},\dots,s_T,a_T)=p(s_1)\prod_{t=1}^T(\pi_\theta(a_t|s_t)p(s_{t+1}|s_t,a_t))$</span></p>
<p>The equation above is the probability of a trajectory.</p>
<p>But in my opinion, the equation should be,
<span class="math-container">$p_{\theta}(s_{1},a_{1},\dots,s_T,a_T)=p(s_1)\prod_{t=1}^T\pi_\theta(a_t|s_t)\prod_{t=1}^{T-1}p(s_{t+1}|s_t,a_t)$</span></p>
<p>I don't know why <span class="math-container">$p(s_{T+1}|s_T,a_T)$</span> emerges in the original equation because there is no <span class="math-container">$s_{T+1},a_{T+1}$</span> in the trajectory. I think I'm missing something.</p>
<p>Can somebody help me?</p>

CC BY-SA 4.0
2022-08-09T01:14:12.327Z
This might be just bayesian factorization. Do you have the image of the trajectory graph?
desert_ranger
CC BY-SA 4.0
2022-08-09T08:04:27.147Z
@desert_ranger Oh I don't have it. But it's just the general trajectory which starts at s_1 and then do the action a_1, so on and forth. I know it's a bayesian factorization, but I still don't know why s_T+1 comes in the first equation.
Feel
<reinforcement-learning><probability>
Question for the derivation of the probability of a trajectory
Feel
# Question for the derivation of the probability of a trajectory

(Asked by: username_0 on 2022-08-06)

I'm studying reinforcement learning now and I'm quite a newbie to this field.
I have some questions about how to derive the equation as below.

$p\_{\theta}(s\_{1},a\_{1},\dots,s\_T,a\_T)=p(s\_1)\prod\_{t=1}^T(\pi\_\theta(a\_t|s\_t)p(s\_{t+1}|s\_t,a\_t))$

The equation above is the probability of a trajectory.

But in my opinion, the equation should be,
$p\_{\theta}(s\_{1},a\_{1},\dots,s\_T,a\_T)=p(s\_1)\prod\_{t=1}^T\pi\_\theta(a\_t|s\_t)\prod\_{t=1}^{T-1}p(s\_{t+1}|s\_t,a\_t)$

I don't know why $p(s\_{T+1}|s\_T,a\_T)$ emerges in the original equation because there is no $s\_{T+1},a\_{T+1}$ in the trajectory. I think I'm missing something.

Can somebody help me?

**Tags:** reinforcement-learning, probability

> **Comments:**
> **username_1**: This might be just bayesian factorization. Do you have the image of the trajectory graph?  
> **username_0**: @username_1 Oh I don't have it. But it's just the general trajectory which starts at s_1 and then do the action a_1, so on and forth. I know it's a bayesian factorization, but I still don't know why s_T+1 comes in the first equation.  

---

## 0 answers

