Answers.Body: <p>If we use <span class="math-container">$T$</span> as the notation for the terminal state, then the last action is <span class="math-container">$a_{T-1}$</span>. This is because when you reach state <span class="math-container">$s_T$</span> you don't take another action, which would be <span class="math-container">$a_T$</span>, because the episode is finished upon reaching the terminal state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-22T10:20:19.950Z
Answers.LastActivityDate: 2021-03-22T10:20:19.950Z
Answers.UserDisplayName: David
Body: <p>From what I learn from CS285 and OpenAI's spinning up, a trajectory in RL is a sequence of state-action pairs:</p>
  <p><span class="math-container">$$\tau = \{s_0, a_0, ..., s_t, a_t\}$$</span></p>
  <p>And the resulting trajectory probability is:</p>
  <p><span class="math-container">$$ P(\tau \mid \pi)=\rho_{0}\left(s_{0}\right) \prod_{t=0}^{T-1} P\left(s_{t+1} \mid s_{t}, a_{t}\right) \pi\left(a_{t} \mid s_{t}\right) $$</span></p>
  <ol>
  <li><p>From CS285: <a href="http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf" rel="nofollow noreferrer">http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf</a></p>
  </li>
  <li><p>From spinning up: <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#trajectories" rel="nofollow noreferrer">https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#trajectories</a></p>
  </li>
  </ol>
  <p>However, from my derivation, the above trajectory probability actually corresponds to the following sequence where the last action <span class="math-container">$a_t$</span> is absent:</p>
  <p><span class="math-container">$$ \tau = \{s_0, a_0, ..., s_t\} $$</span></p>
  <p>Can someone please help me clarify this confusion?</p>
  
Tags: <reinforcement-learning><terminology><notation>
Title: Does a trajectory in reinforcement learning contain the last action?
UserDisplayName: Ezekiel Chen
ThreadText: # Does a trajectory in reinforcement learning contain the last action?
  
  (Asked by: username_0 on 2021-03-22)
  
  From what I learn from CS285 and OpenAI's spinning up, a trajectory in RL is a sequence of state-action pairs:
  
  $$\tau = \{s\_0, a\_0, ..., s\_t, a\_t\}$$
  
  And the resulting trajectory probability is:
  
  $$ P(\tau \mid \pi)=\rho\_{0}\left(s\_{0}\right) \prod\_{t=0}^{T-1} P\left(s\_{t+1} \mid s\_{t}, a\_{t}\right) \pi\left(a\_{t} \mid s\_{t}\right) $$
  
  1. From CS285: <http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf>
  2. From spinning up: <https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#trajectories>
  
  However, from my derivation, the above trajectory probability actually corresponds to the following sequence where the last action $a\_t$ is absent:
  
  $$ \tau = \{s\_0, a\_0, ..., s\_t\} $$
  
  Can someone please help me clarify this confusion?
  
  **Tags:** reinforcement-learning, terminology, notation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-22. Score: 1
  
  If we use $T$ as the notation for the terminal state, then the last action is $a\_{T-1}$. This is because when you reach state $s\_T$ you don't take another action, which would be $a\_T$, because the episode is finished upon reaching the terminal state.
  
  
  ---
  
  