Answers.Body: <p>One way to measure the distance between two policies is to use the <strong>Kullback-Leibler (KL) divergence</strong>. This measures the amount of information lost when one policy is used to approximate the other.</p>
  <p>Another way to measure is to use the <strong>Wasserstein metric</strong>. This measures the amount of &quot;work&quot; or &quot;energy&quot; required to transform one policy into the other.</p>
  <p>A third way is to use the <strong>Mutual Information (MI) metric</strong>. This measures the amount of information shared between the two policies.</p>
  <blockquote>
  <p>yes! <strong>Euclidean distance</strong> would way to measure the distance be between the two policies action vectors</p>
  </blockquote>
  <p>It does, sort of. The idea is that you have a reward function that encodes the task you want to solve. The agent is trying to find an optimal policy that maximizes this reward function. In order to do this, the agent must learn the reward function.</p>
  <p>One way to think about this is that the agent is trying to find a mapping from states to actions that will maximise the reward. This is the same as finding a mapping from states to a goal state that maximises the reward. So, if you have two different agents, they will both be trying to find a mapping from states to actions that maximises the reward. However, they will likely find different mappings, because they have different starting</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-16T11:46:58.080Z
Answers.Text: OP’s question was regarding deterministic policies. I’m not sure how useful KLD is in this instance as they are trivially dissimilar (I think, happy to be told otherwise).
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T10:32:34.687Z
Answers.LastActivityDate: 2022-10-17T10:32:34.687Z
Answers.UserDisplayName: Faizy
Body: <p><a href="https://ai.stackexchange.com/questions/25166/are-there-some-notions-of-distance-between-two-policies">This question</a> asks if there is a way to measure distance between policies that are in fact probability distributions.</p>
  <p><strong>In the case of continuous control with <strong>deterministic</strong> policies where they take a <strong>state as input</strong> and <strong>return an action vector</strong>, what would be the best method to measure how close two policies are from each other?</strong></p>
  <p>A naive approach that came to my mind would be to:</p>
  <ul>
  <li>Run both policies A and B to produce a trajectory each and record all states visited.</li>
  <li>For each state encountered by policy A, ask policy B which action it would take (and vice-versa). Hence we would have, for every state encountered, both A and B action vectors.</li>
  <li>For each state, compare action vectors of A and B by using a common distance (Euclidean distance?)</li>
  <li>Take the average (maybe maximum) of those distances.</li>
  </ul>
  <p>Does it make any sense from a theoretical point of view?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-31T12:49:21.233Z
Comments.Text: Can you clarify what the action vector produced by the policy actually represents? If it's not a probability distribution, then why do you have an action vector? Is it a policy that produces multiple actions, all of them need to be taken at time step $t$? Is this a multi-agent system? Please, edit your post to clarify this.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><policies><metric><kl-divergence>
Title: What is the best distance measure between policies that are not probability distributions?
UserDisplayName: Valentin Macé
ThreadText: # What is the best distance measure between policies that are not probability distributions?
  
  (Asked by: username_0 on 2022-10-04)
  
  [This question](https://ai.stackexchange.com/questions/25166/are-there-some-notions-of-distance-between-two-policies) asks if there is a way to measure distance between policies that are in fact probability distributions.
  
  **In the case of continuous control with **deterministic** policies where they take a **state as input** and **return an action vector**, what would be the best method to measure how close two policies are from each other?**
  
  A naive approach that came to my mind would be to:
  
  * Run both policies A and B to produce a trajectory each and record all states visited.
  * For each state encountered by policy A, ask policy B which action it would take (and vice-versa). Hence we would have, for every state encountered, both A and B action vectors.
  * For each state, compare action vectors of A and B by using a common distance (Euclidean distance?)
  * Take the average (maybe maximum) of those distances.
  
  Does it make any sense from a theoretical point of view?
  
  **Tags:** reinforcement-learning, policies, metric, kl-divergence
  > **Comments:**
  > **username_1**: Can you clarify what the action vector produced by the policy actually represents? If it's not a probability distribution, then why do you have an action vector? Is it a policy that produces multiple actions, all of them need to be taken at time step $t$? Is this a multi-agent system? Please, edit your post to clarify this.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-17. Score: 0
  
  One way to measure the distance between two policies is to use the **Kullback-Leibler (KL) divergence**. This measures the amount of information lost when one policy is used to approximate the other.
  
  Another way to measure is to use the **Wasserstein metric**. This measures the amount of "work" or "energy" required to transform one policy into the other.
  
  A third way is to use the **Mutual Information (MI) metric**. This measures the amount of information shared between the two policies.
  
  > yes! **Euclidean distance** would way to measure the distance be between the two policies action vectors
  
  It does, sort of. The idea is that you have a reward function that encodes the task you want to solve. The agent is trying to find an optimal policy that maximizes this reward function. In order to do this, the agent must learn the reward function.
  
  One way to think about this is that the agent is trying to find a mapping from states to actions that will maximise the reward. This is the same as finding a mapping from states to a goal state that maximises the reward. So, if you have two different agents, they will both be trying to find a mapping from states to actions that maximises the reward. However, they will likely find different mappings, because they have different starting
  
  > **Comments:**
  > **username_3**: OP’s question was regarding deterministic policies. I’m not sure how useful KLD is in this instance as they are trivially dissimilar (I think, happy to be told otherwise).  
  
  ---
  
  