Answers.Body: <p>I don't really think there are fixed, different definitions for all those terms that everyone agrees upon. In most contexts, they're going to be quite interchangeable, and if anyone is really using them in a context where they are supposed to have crucially important, different meanings, they should probably precisely define them right there.</p>
  <hr />
  <blockquote>
  <p>I think episode has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of <span class="math-container">$(s,a,r)$</span> sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.</p>
  </blockquote>
  <p>Agreed with this.</p>
  <blockquote>
  <p>With trajectory, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.</p>
  </blockquote>
  <p>I can't really think of cases where it's sensible to talk about trajectories with tuples shuffled into an arbitrary order. I'd still think of trajectories as having to be in the &quot;correct&quot; order in which they were experienced. But I do agree that trajectories can be little samples (for instance, little sequences of experience that we store in an experience replay buffer). So, every full episode would be a (long) trajectory, but not every trajectory is a full episode (a trajectory can just be a small part of an episode).</p>
  <blockquote>
  <p>I think rollout is somewhere in between since I commonly see it used to refer to a sampled sequence of <span class="math-container">$(s, a,r)$</span> from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.</p>
  </blockquote>
  <p>I'd say that... often a rollout should have a &quot;terminal&quot; state as ending, but maybe not a true &quot;initial&quot; state of an episode as the start. We might be in the middle of an episode, and then say that we &quot;roll out&quot;, which to me implies that we keep going until the end of an episode. I don't think this term is as common as the other two in Reinforcement Learning, but more common in search/planning literature (in particular, Monte Caro Tree Search).</p>
  <p>That said, when I'm working with MCTS I often like to put a limit on my rollouts where I cut them off if no terminal state was reached yet... so that isn't exactly a crisp definition either.</p>
  <p>Due to how commonly-used this term is specifically in MCTS, and other Monte-Carlo-based algorithms, I also associate a greater degree of randomness with the term &quot;rollout&quot;. When I hear &quot;episode&quot; or &quot;trajectory&quot;, I can envision a highly sophisticated, &quot;intelligent&quot; policy being used to select actions, but when I hear &quot;rollout&quot; I am inclined to think of a greater degree of randomness being incorporated in the action selection (maybe uniformly random, or maybe with some cheap-to-compute, simple policy for biasing away from uniformity). Again, that's really just an association I have in my mind with the term and not a crisp definition.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-08T18:01:07.203Z
Answers.Text: So is it okay to say that all three of them represent a sequence of (s, a, r) tuples, where an episode is generally from an initial state to a terminal state, a rollout is from an initial or intermediate state to a terminal state, and a trajectory is between any two states?
Answers.UserDisplayName: nikhilweee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-08T19:17:31.147Z
Answers.Text: @nikhilweee That's how I (and I guess many people familiar with RL literature) would generally interpret them if no further clarification were given, yes.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-15T20:11:05.780Z
Answers.LastActivityDate: 2022-01-08T11:37:25.407Z
Answers.LastEditDate: 2022-01-08T11:37:25.407Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I often see the terms episode, trajectory, and rollout to refer to basically the same thing, a list of (state, action, rewards). Are there any concrete differences between the terms or can they be used interchangeably?</p>
  <p>In the following paragraphs, I'll summarize my current slightly vague understanding of the terms. Please point any inaccuracy or missing details in my definitions.</p>
  <p>I think <em>episode</em> has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of <span class="math-container">$(s, a, r)$</span> sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.</p>
  <p>With <em>trajectory</em>, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.</p>
  <p>I think <em>rollout</em> is somewhere in between since I commonly see it used to refer to a sampled sequence of <span class="math-container">$(s, a, r)$</span> from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.</p>
  
Tags: <reinforcement-learning><terminology><comparison>
Title: What is the difference between an episode, a trajectory and a rollout?
UserDisplayName: Paula Vega
ThreadText: # What is the difference between an episode, a trajectory and a rollout?
  
  (Asked by: username_0 on 2019-02-15)
  
  I often see the terms episode, trajectory, and rollout to refer to basically the same thing, a list of (state, action, rewards). Are there any concrete differences between the terms or can they be used interchangeably?
  
  In the following paragraphs, I'll summarize my current slightly vague understanding of the terms. Please point any inaccuracy or missing details in my definitions.
  
  I think *episode* has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of $(s, a, r)$ sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.
  
  With *trajectory*, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.
  
  I think *rollout* is somewhere in between since I commonly see it used to refer to a sampled sequence of $(s, a, r)$ from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.
  
  **Tags:** reinforcement-learning, terminology, comparison
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-15. Score: 5
  
  I don't really think there are fixed, different definitions for all those terms that everyone agrees upon. In most contexts, they're going to be quite interchangeable, and if anyone is really using them in a context where they are supposed to have crucially important, different meanings, they should probably precisely define them right there.
  
  ---
  
  > I think episode has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of $(s,a,r)$ sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.
  
  Agreed with this.
  
  > With trajectory, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.
  
  I can't really think of cases where it's sensible to talk about trajectories with tuples shuffled into an arbitrary order. I'd still think of trajectories as having to be in the "correct" order in which they were experienced. But I do agree that trajectories can be little samples (for instance, little sequences of experience that we store in an experience replay buffer). So, every full episode would be a (long) trajectory, but not every trajectory is a full episode (a trajectory can just be a small part of an episode).
  
  > I think rollout is somewhere in between since I commonly see it used to refer to a sampled sequence of $(s, a,r)$ from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.
  
  I'd say that... often a rollout should have a "terminal" state as ending, but maybe not a true "initial" state of an episode as the start. We might be in the middle of an episode, and then say that we "roll out", which to me implies that we keep going until the end of an episode. I don't think this term is as common as the other two in Reinforcement Learning, but more common in search/planning literature (in particular, Monte Caro Tree Search).
  
  That said, when I'm working with MCTS I often like to put a limit on my rollouts where I cut them off if no terminal state was reached yet... so that isn't exactly a crisp definition either.
  
  Due to how commonly-used this term is specifically in MCTS, and other Monte-Carlo-based algorithms, I also associate a greater degree of randomness with the term "rollout". When I hear "episode" or "trajectory", I can envision a highly sophisticated, "intelligent" policy being used to select actions, but when I hear "rollout" I am inclined to think of a greater degree of randomness being incorporated in the action selection (maybe uniformly random, or maybe with some cheap-to-compute, simple policy for biasing away from uniformity). Again, that's really just an association I have in my mind with the term and not a crisp definition.
  
  > **Comments:**
  > **username_2**: So is it okay to say that all three of them represent a sequence of (s, a, r) tuples, where an episode is generally from an initial state to a terminal state, a rollout is from an initial or intermediate state to a terminal state, and a trajectory is between any two states?  
  > **username_1**: @nikhilweee That's how I (and I guess many people familiar with RL literature) would generally interpret them if no further clarification were given, yes.  
  
  ---
  
  