Answers.Body: <p>Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance are not necessarily <a href="https://en.wikipedia.org/wiki/Metric_(mathematics)" rel="noreferrer">metrics</a> in a mathematical sense).</p>
  <p>A common measure is the <a href="https://en.wikipedia.org/wiki/Relative_entropy" rel="noreferrer"><strong>Kullback–Leibler divergence</strong></a> (which is not a metric, in a mathematical sense, given that it does not satisfy certain required conditions for being a metric). For example, in section 4 of <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="noreferrer">the PPO paper</a>, the KL divergence is used as a regulariser (which is actually quite common, for instance, in the context of variational Bayesian neural networks). The <a href="http://proceedings.mlr.press/v37/schulman15.pdf" rel="noreferrer">TRPO</a> also uses the KL divergence.</p>
  <p>The <a href="https://en.wikipedia.org/wiki/Wasserstein_metric" rel="noreferrer"><strong>Wasserstein metric</strong></a> has also been used in RL, for instance, in <a href="https://arxiv.org/pdf/1707.06887.pdf" rel="noreferrer">distributional RL</a> (but, in this case, not to compare policies but distributions over value functions).</p>
  <p>You can find more info about statistical distances <a href="https://en.wikipedia.org/wiki/Statistical_distance" rel="noreferrer">here</a>. The specific distance that you use may depend on the problem that you want to solve and the properties that you want your distance to have. For example, the KL divergence is unbounded above, so, if that's not desirable, you could choose another one. The paper <a href="https://arxiv.org/pdf/math/0209021.pdf" rel="noreferrer">On choosing and bounding probability metrics</a> (2002, by Gibbs and Su) may also be useful. <a href="https://ai.stackexchange.com/a/23952/2444">Here</a> I also talk about the KL divergence and total variation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T17:06:15.257Z
Answers.Text: Worth noting the Wasserstein metric requires there to be metric over the term that the distribution applies to, so may not apply well to discrete action spaces where there is no conceptual space or ordering between terms (e.g. what is the distance between "up" and "left" actions on a game console?)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T17:11:55.630Z
Answers.Text: @NeilSlater Thanks for pointing this out. Right now, I am only familiar with the details of the KL divergence, although, as I say in this answer, I know that the Wasserstein metric has also been used in RL and I would guess that other measures of distance have been used. If you are familiar with the details of other measures (other than the KL divergence, which is quite popular), I think it may be worth providing another answer with another perspective, which maybe provides some guidance on how to choose it in the context of RL.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T17:01:51.387Z
Answers.LastActivityDate: 2020-12-13T17:01:51.387Z
Answers.UserDisplayName: nbro
Body: <p>I want to determine some distance between two policies <span class="math-container">$\pi_1 (a \mid s)$</span> and <span class="math-container">$\pi_2 (a \mid s)$</span>, i.e. something like <span class="math-container">$\vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert$</span>, where <span class="math-container">$\pi_i (a\mid s)$</span> is the vector <span class="math-container">$(\pi_i (a_1 \mid s), \dots, \pi_i(a_n \mid s))$</span>. I am looking for a sensible notion for such a distance.</p>
  <p>Are there some standard norms/metrics used in the literature for determining a distance between policies?</p>
  
Tags: <reinforcement-learning><reference-request><policies><kl-divergence><wasserstein-metric>
Title: Are there some notions of distance between two policies?
UserDisplayName: Felix P.
ThreadText: # Are there some notions of distance between two policies?
  
  (Asked by: username_0 on 2020-12-13)
  
  I want to determine some distance between two policies $\pi\_1 (a \mid s)$ and $\pi\_2 (a \mid s)$, i.e. something like $\vert \vert \pi\_1 (a \mid s) - \pi\_2(a \mid s) \vert \vert$, where $\pi\_i (a\mid s)$ is the vector $(\pi\_i (a\_1 \mid s), \dots, \pi\_i(a\_n \mid s))$. I am looking for a sensible notion for such a distance.
  
  Are there some standard norms/metrics used in the literature for determining a distance between policies?
  
  **Tags:** reinforcement-learning, reference-request, policies, kl-divergence, wasserstein-metric
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-13. Score: 5 | Accepted answer
  
  Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance are not necessarily [metrics](https://en.wikipedia.org/wiki/Metric_(mathematics)) in a mathematical sense).
  
  A common measure is the [**Kullback–Leibler divergence**](https://en.wikipedia.org/wiki/Relative_entropy) (which is not a metric, in a mathematical sense, given that it does not satisfy certain required conditions for being a metric). For example, in section 4 of [the PPO paper](https://arxiv.org/pdf/1707.06347.pdf), the KL divergence is used as a regulariser (which is actually quite common, for instance, in the context of variational Bayesian neural networks). The [TRPO](http://proceedings.mlr.press/v37/schulman15.pdf) also uses the KL divergence.
  
  The [**Wasserstein metric**](https://en.wikipedia.org/wiki/Wasserstein_metric) has also been used in RL, for instance, in [distributional RL](https://arxiv.org/pdf/1707.06887.pdf) (but, in this case, not to compare policies but distributions over value functions).
  
  You can find more info about statistical distances [here](https://en.wikipedia.org/wiki/Statistical_distance). The specific distance that you use may depend on the problem that you want to solve and the properties that you want your distance to have. For example, the KL divergence is unbounded above, so, if that's not desirable, you could choose another one. The paper [On choosing and bounding probability metrics](https://arxiv.org/pdf/math/0209021.pdf) (2002, by Gibbs and Su) may also be useful. [Here](https://ai.stackexchange.com/a/23952/2444) I also talk about the KL divergence and total variation.
  
  > **Comments:**
  > **username_2**: Worth noting the Wasserstein metric requires there to be metric over the term that the distribution applies to, so may not apply well to discrete action spaces where there is no conceptual space or ordering between terms (e.g. what is the distance between "up" and "left" actions on a game console?)  
  > **username_1**: @username_2 Thanks for pointing this out. Right now, I am only familiar with the details of the KL divergence, although, as I say in this answer, I know that the Wasserstein metric has also been used in RL and I would guess that other measures of distance have been used. If you are familiar with the details of other measures (other than the KL divergence, which is quite popular), I think it may be worth providing another answer with another perspective, which maybe provides some guidance on how to choose it in the context of RL.  
  
  ---
  
  