Answers.Body: <p>Neil’s answer is good but I’m observing that strictly translating the following Quora’s answer statement </p>
  
  <blockquote>
    <p>In reinforcement learning terminology, a trajectory <span class="math-container">$\tau$</span> is the path of the agent through the state space up until the horizon </p>
  </blockquote>
  
  <p>we get <span class="math-container">$\tau = \{ s_{t} \}_{t \in [t_{0}, t_{H}]} \quad s_{t} \in \mathcal{S}$</span> with </p>
  
  <ul>
  <li><span class="math-container">$\mathcal{S}$</span> the State Space </li>
  <li><span class="math-container">$t_{0}$</span> initial time </li>
  <li><span class="math-container">$t_{H} &gt; t_{0}$</span> time associated to a certain event <span class="math-container">$H$</span></li>
  </ul>
  
  <p>So according to Quora's answer author, it should just be a temporal sequence of states (no action and rewards)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-13T15:46:14.873Z
Answers.LastActivityDate: 2018-11-13T15:46:14.873Z
Answers.UserDisplayName: Nicola Bernini
Answers.Body: <p>A trajectory ist just a <strong>sequence of states and actions</strong>. In RL, the goal is to maximize the reward, by finding the right trajectories.</p>
  
  <p><span class="math-container">$$
  \operatorname{max}_\tau R(\tau)
  $$</span></p>
  
  <p>This means
  maximizing not immediate reward (caused by one action from a state), but cumulative reward (all states and actions: trajectory)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T19:34:37.757Z
Answers.LastActivityDate: 2020-01-10T10:17:31.147Z
Answers.LastEditDate: 2020-01-10T10:17:31.147Z
Answers.UserDisplayName: Justin Lange
Answers.Body: <p>In answer that you linked, I may have used an informal definition of "trajectory", but essentially the same thing as the quote. A "trajectory" is the sequence of what has happened (in terms of state, action, reward) over a set of contiguous timestamps, from a single episode, or a single part of a continuous problem.</p>
  
  <p>So <span class="math-container">$(s_3, a_3, r_4, s_4, a_4, r_5, s_5, a_5, r_6, s_6)$</span> taken from any scenario where an agent was used in the problem environment would be a trajectory - at least as I intended it in the answer. This could be from real-world data, or a simulation. It could involve a totally random or untrained agent, or a fully-optimised policy.</p>
  
  <p>In the other definition that you have found, the focus on states and a <em>horizon</em> could make it slightly different, but actually I suspect that it is the same thing, as it is not that useful to only know the states. The Quora answer is probably just using "path of the agent through the state space" as shorthand to describe the same data.</p>
  
  <p>A "horizon" in reinforcement learning is a future point relative to a time step, beyond which you do not care about reward (so you sum the rewards from time <span class="math-container">$t$</span> to <span class="math-container">$t+H$</span>). Fixed horizons can be used as an alternative to a discount factor for limiting sums of reward in continuous problems. They may also be used in other approaches, but basically mean the same thing - a time step beyond which you don't account for what happens.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-01T09:18:53.140Z
Answers.Text: Thanks and just to confirm - If the "trajectory" itself means the list or array, in what you wrote in the answer (`The de-correlation effect is more important than following **sequence of trajectories**`) the word *sequence* is redundant, since the trajctory itself is sequence? Or not?
Answers.UserDisplayName: Blaszard
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-01T13:51:33.410Z
Answers.Text: @Baszard It's partially redundant yes. But in theory you *could* sample from a trajectory at random.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-31T15:13:49.273Z
Answers.LastActivityDate: 2018-11-12T20:39:24.003Z
Answers.LastEditDate: 2018-11-12T20:39:24.003Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm now learning about reinforcement learning, but I just found the word "trajectory" in <a href="https://datascience.stackexchange.com/a/24924/8432">this answer</a>.</p>
  
  <p>However, I'm not sure what it means. I read a few books on the Reinforcement Learning but none of them mentioned it. Usually these introductionary books mention agent, environment, action, policy, and reward, but not "trajectory".</p>
  
  <p>So, what does it mean? 
  According to <a href="https://www.quora.com/In-the-context-of-reinforcement-learning-what-is-the-difference-between-a-trajectory-and-a-policy-Also-what-is-the-difference-between-trajectory-optimization-and-policy-optimization" rel="noreferrer">this answer</a> over Quora:</p>
  
  <blockquote>
    <p>In reinforcement learning terminology, a trajectory <span class="math-container">$\tau$</span> is the path of the agent through the state space up until the horizon <span class="math-container">$H$</span>. The goal of an on-policy algorithm is to maximize the expected reward of the agent over trajectories.</p>
  </blockquote>
  
  <p>Does it mean that the "trajectory" is the total path from the current state the agent is in to the final state (terminal state) that the episode finishes at? Or is it something else? (I'm not sure what the "horizon" mean, either).</p>
  
Tags: <reinforcement-learning><terminology>
Title: What is a "trajectory" in reinforcement learning?
UserDisplayName: Blaszard
ThreadText: # What is a "trajectory" in reinforcement learning?
  
  (Asked by: username_0 on 2018-07-31)
  
  I'm now learning about reinforcement learning, but I just found the word "trajectory" in [this answer](https://datascience.stackexchange.com/a/24924/8432).
  
  However, I'm not sure what it means. I read a few books on the Reinforcement Learning but none of them mentioned it. Usually these introductionary books mention agent, environment, action, policy, and reward, but not "trajectory".
  
  So, what does it mean?
  According to [this answer](https://www.quora.com/In-the-context-of-reinforcement-learning-what-is-the-difference-between-a-trajectory-and-a-policy-Also-what-is-the-difference-between-trajectory-optimization-and-policy-optimization) over Quora:
  
  > In reinforcement learning terminology, a trajectory $\tau$ is the path of the agent through the state space up until the horizon $H$. The goal of an on-policy algorithm is to maximize the expected reward of the agent over trajectories.
  
  Does it mean that the "trajectory" is the total path from the current state the agent is in to the final state (terminal state) that the episode finishes at? Or is it something else? (I'm not sure what the "horizon" mean, either).
  
  **Tags:** reinforcement-learning, terminology
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2018-07-31. Score: 14 | Accepted answer
  
  In answer that you linked, I may have used an informal definition of "trajectory", but essentially the same thing as the quote. A "trajectory" is the sequence of what has happened (in terms of state, action, reward) over a set of contiguous timestamps, from a single episode, or a single part of a continuous problem.
  
  So $(s\_3, a\_3, r\_4, s\_4, a\_4, r\_5, s\_5, a\_5, r\_6, s\_6)$ taken from any scenario where an agent was used in the problem environment would be a trajectory - at least as I intended it in the answer. This could be from real-world data, or a simulation. It could involve a totally random or untrained agent, or a fully-optimised policy.
  
  In the other definition that you have found, the focus on states and a *horizon* could make it slightly different, but actually I suspect that it is the same thing, as it is not that useful to only know the states. The Quora answer is probably just using "path of the agent through the state space" as shorthand to describe the same data.
  
  A "horizon" in reinforcement learning is a future point relative to a time step, beyond which you do not care about reward (so you sum the rewards from time $t$ to $t+H$). Fixed horizons can be used as an alternative to a discount factor for limiting sums of reward in continuous problems. They may also be used in other approaches, but basically mean the same thing - a time step beyond which you don't account for what happens.
  
  > **Comments:**
  > **username_0**: Thanks and just to confirm - If the "trajectory" itself means the list or array, in what you wrote in the answer (`The de-correlation effect is more important than following **sequence of trajectories**`) the word *sequence* is redundant, since the trajctory itself is sequence? Or not?  
  > **username_3**: @Baszard It's partially redundant yes. But in theory you *could* sample from a trajectory at random.  
  
  ---
  
  ### Answer by username_1 on 2018-11-13. Score: 3
  
  Neil’s answer is good but I’m observing that strictly translating the following Quora’s answer statement
  
  > In reinforcement learning terminology, a trajectory $\tau$ is the path of the agent through the state space up until the horizon
  
  we get $\tau = \{ s\_{t} \}\_{t \in [t\_{0}, t\_{H}]} \quad s\_{t} \in \mathcal{S}$ with
  
  * $\mathcal{S}$ the State Space
  * $t\_{0}$ initial time
  * $t\_{H} > t\_{0}$ time associated to a certain event $H$
  
  So according to Quora's answer author, it should just be a temporal sequence of states (no action and rewards)
  
  
  ---
  
  ### Answer by username_2 on 2020-01-09. Score: 3
  
  A trajectory ist just a **sequence of states and actions**. In RL, the goal is to maximize the reward, by finding the right trajectories.
  
  $$
  \operatorname{max}\_\tau R(\tau)
  $$
  
  This means
  maximizing not immediate reward (caused by one action from a state), but cumulative reward (all states and actions: trajectory)
  
  
  ---
  
  