Body: <p>I'm studying reinforcement learning now and I'm quite a newbie to this field.
  I have some questions about how to derive the equation as below.</p>
  <p><span class="math-container">$p_{\theta}(s_{1},a_{1},\dots,s_T,a_T)=p(s_1)\prod_{t=1}^T(\pi_\theta(a_t|s_t)p(s_{t+1}|s_t,a_t))$</span></p>
  <p>The equation above is the probability of a trajectory.</p>
  <p>But in my opinion, the equation should be,
  <span class="math-container">$p_{\theta}(s_{1},a_{1},\dots,s_T,a_T)=p(s_1)\prod_{t=1}^T\pi_\theta(a_t|s_t)\prod_{t=1}^{T-1}p(s_{t+1}|s_t,a_t)$</span></p>
  <p>I don't know why <span class="math-container">$p(s_{T+1}|s_T,a_T)$</span> emerges in the original equation because there is no <span class="math-container">$s_{T+1},a_{T+1}$</span> in the trajectory. I think I'm missing something.</p>
  <p>Can somebody help me?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-09T01:14:12.327Z
Comments.Text: This might be just bayesian factorization. Do you have the image of the trajectory graph?
Comments.UserDisplayName: desert_ranger
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-09T08:04:27.147Z
Comments.Text: @desert_ranger Oh I don't have it. But it's just the general trajectory which starts at s_1 and then do the action a_1, so on and forth. I know it's a bayesian factorization, but I still don't know why s_T+1 comes in the first equation.
Comments.UserDisplayName: Feel
Tags: <reinforcement-learning><probability>
Title: Question for the derivation of the probability of a trajectory
UserDisplayName: Feel
ThreadText: # Question for the derivation of the probability of a trajectory
  
  (Asked by: username_0 on 2022-08-06)
  
  I'm studying reinforcement learning now and I'm quite a newbie to this field.
  I have some questions about how to derive the equation as below.
  
  $p\_{\theta}(s\_{1},a\_{1},\dots,s\_T,a\_T)=p(s\_1)\prod\_{t=1}^T(\pi\_\theta(a\_t|s\_t)p(s\_{t+1}|s\_t,a\_t))$
  
  The equation above is the probability of a trajectory.
  
  But in my opinion, the equation should be,
  $p\_{\theta}(s\_{1},a\_{1},\dots,s\_T,a\_T)=p(s\_1)\prod\_{t=1}^T\pi\_\theta(a\_t|s\_t)\prod\_{t=1}^{T-1}p(s\_{t+1}|s\_t,a\_t)$
  
  I don't know why $p(s\_{T+1}|s\_T,a\_T)$ emerges in the original equation because there is no $s\_{T+1},a\_{T+1}$ in the trajectory. I think I'm missing something.
  
  Can somebody help me?
  
  **Tags:** reinforcement-learning, probability
  
  > **Comments:**
  > **username_1**: This might be just bayesian factorization. Do you have the image of the trajectory graph?  
  > **username_0**: @username_1 Oh I don't have it. But it's just the general trajectory which starts at s_1 and then do the action a_1, so on and forth. I know it's a bayesian factorization, but I still don't know why s_T+1 comes in the first equation.  
  
  ---
  
  ## 0 answers
  
  