Body: <p>I realize that my question is a bit fuzzy and I am sorry for that. If needed, I will try to make it more rigorous and precice.</p>
  <p>Let <span class="math-container">$\mathcal{M}$</span> be a Markov Decision Process, with state space <span class="math-container">$\mathcal{S}$</span> and action space <span class="math-container">$\mathcal{A}$</span>.
  Let <span class="math-container">$\tau = (s_0, a_0, s_1, a_1, s_2, a_2, \dots)$</span> and <span class="math-container">$\tau' = (s_0', a_0', s_1', a_1', s_2', a_2', \dots)$</span> be two trajectories produced by an agent during two different episodes.</p>
  <p><strong>Question</strong>: Is there any standard way in the Reinforcement Learning literature to compare <span class="math-container">$\tau$</span> and <span class="math-container">$\tau'$</span>? Ideally I am interested in finding a &quot;distance&quot; (it does not need to be a distance in the mathematical sense) <span class="math-container">$d(\tau, \tau ')$</span> such that it reflects the &quot;distance&quot; between the poicies that generated <span class="math-container">$\tau$</span> and <span class="math-container">$\tau'$</span>.</p>
  <p>For example it would be nice if <span class="math-container">$d(\tau, \tau')$</span> would be a good estimator of the KL divergence of <span class="math-container">$\pi$</span> and <span class="math-container">$\pi'$</span>, where <span class="math-container">$\pi$</span> is the policy that generated <span class="math-container">$\tau$</span> and <span class="math-container">$\pi'$</span> the policy that generated <span class="math-container">$\tau'$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-25T15:41:54.363Z
Comments.Text: in what regard do you need to compare two trajectories? maybe you want to compare the "agents" responsible for those two trajectories? then you can calculate the probability ratio of actions between two agents. it is actually used in "importance sampling" when doing off-policy optimization, measuring difference between behaviour and target policy
Comments.UserDisplayName: Alireza
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-25T16:58:37.537Z
Comments.Text: So, ideally I would like to compare the probabilities (policies) of the two agents, for example using the KL divergence (wich is the expectation w.r.t one of the two probabilities of the log of the ratio). However I do not have access to the policies but only to the trajectories. I would like to measure how different they are.
Comments.UserDisplayName: Onil90
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-26T05:26:39.357Z
Comments.Text: Do you only have two trajectories? Do you have access to the MDP model (i.e. transition probabilities)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-26T10:39:24.697Z
Comments.Text: If the goal is to compare agents, then difference in states should not be in the compared vectors to begin with. I'd only compare $\pi(a|s)$ as this represents the agents. $p(s|a)$ part is an attribute of the environment. you can possibly learn $\pi(a|s)$ from data (not to mention how super hard it might be given the number actions or continuity of action space), then you can compare the learned $\pi$ densities, e.g. via KL.
  
  still, as I and also @Onil90 mentioned, the straight forward scenario is having access to transition probs. without that, don't know if there's a standard solution
Comments.UserDisplayName: Alireza
Tags: <reinforcement-learning><markov-decision-process><kl-divergence>
Title: How to compare different trajecories in a Markov Decision Process
UserDisplayName: Onil90
ThreadText: # How to compare different trajecories in a Markov Decision Process
  
  (Asked by: username_0 on 2022-08-25)
  
  I realize that my question is a bit fuzzy and I am sorry for that. If needed, I will try to make it more rigorous and precice.
  
  Let $\mathcal{M}$ be a Markov Decision Process, with state space $\mathcal{S}$ and action space $\mathcal{A}$.
  Let $\tau = (s\_0, a\_0, s\_1, a\_1, s\_2, a\_2, \dots)$ and $\tau' = (s\_0', a\_0', s\_1', a\_1', s\_2', a\_2', \dots)$ be two trajectories produced by an agent during two different episodes.
  
  **Question**: Is there any standard way in the Reinforcement Learning literature to compare $\tau$ and $\tau'$? Ideally I am interested in finding a "distance" (it does not need to be a distance in the mathematical sense) $d(\tau, \tau ')$ such that it reflects the "distance" between the poicies that generated $\tau$ and $\tau'$.
  
  For example it would be nice if $d(\tau, \tau')$ would be a good estimator of the KL divergence of $\pi$ and $\pi'$, where $\pi$ is the policy that generated $\tau$ and $\pi'$ the policy that generated $\tau'$.
  
  **Tags:** reinforcement-learning, markov-decision-process, kl-divergence
  > **Comments:**
  > **username_1**: in what regard do you need to compare two trajectories? maybe you want to compare the "agents" responsible for those two trajectories? then you can calculate the probability ratio of actions between two agents. it is actually used in "importance sampling" when doing off-policy optimization, measuring difference between behaviour and target policy  
  > **username_0**: So, ideally I would like to compare the probabilities (policies) of the two agents, for example using the KL divergence (wich is the expectation w.r.t one of the two probabilities of the log of the ratio). However I do not have access to the policies but only to the trajectories. I would like to measure how different they are.  
  > **username_2**: Do you only have two trajectories? Do you have access to the MDP model (i.e. transition probabilities)?  
  > **username_1**: If the goal is to compare agents, then difference in states should not be in the compared vectors to begin with. I'd only compare $\pi(a|s)$ as this represents the agents. $p(s|a)$ part is an attribute of the environment. you can possibly learn $\pi(a|s)$ from data (not to mention how super hard it might be given the number actions or continuity of action space), then you can compare the learned $\pi$ densities, e.g. via KL.  still, as I and also @Onil90 mentioned, the straight forward scenario is having access to transition probs. without that, don't know if there's a standard solution  
  
  ---
  
  ## 0 answers
  
  