Answers.Body: <p>&quot;Metric&quot; should be understood as &quot;a function of the trained model and of a dataset which returns a number&quot;.</p>
  <p>For example, in reinforcement learning, one can use as an evaluation metric the total cumulative discounted reward that a policy gets on a set of episodes for a given task, starting from the same initial state <span class="math-container">$S_0$</span>.</p>
  <p>That's a function <span class="math-container">$f(\tau_1, \tau_2, ... | \pi, S_0)$</span>, where <span class="math-container">$\tau_1$</span> is the first trajectory obtained by running policy <span class="math-container">$\pi$</span> on the task, <span class="math-container">$\tau_2$</span> the second one, etc. (we assume that <span class="math-container">$\pi$</span> is stochastic, otherwise it doesn't make sense to run multiple episodes). <span class="math-container">$f$</span> is an &quot;evaluation metric&quot; but is certainly not a distance function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-16T19:20:23.033Z
Answers.LastActivityDate: 2022-05-16T19:20:23.033Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>Consider the following abstract from the research paper titled <a href="https://arxiv.org/pdf/1801.01973.pdf" rel="nofollow noreferrer">A Note on the Inception Score</a> for instance</p>
  <blockquote>
  <p>Deep generative models are powerful tools that have produced
  impressive results in recent years. These advances have been for the
  most part empirically driven, making it essential that we use
  high-quality <strong>evaluation metrics</strong>. In this paper, we provide new
  insights into the Inception Score, a recently proposed and widely used
  <strong>evaluation metric</strong> for generative models, and demonstrate that it fails
  to provide useful guidance when comparing models. We discuss both
  suboptimalities of the metric itself and issues with its application.
  Finally, we call for researchers to be more systematic and careful
  when evaluating and comparing generative models, as the advancement of
  the field depends upon it.</p>
  </blockquote>
  <p>Here we can observe the usage of word metric several times. In mathematics, the word metric is used only in the context of metric spaces afaik. The <a href="http://www-groups.mcs.st-andrews.ac.uk/%7Ejohn/MT4522/Lectures/L5.html#:%7E:text=A%20metric%20space%20is%20a,(x%2C%20y)%20%3D%200" rel="nofollow noreferrer">definition for metric space</a> and metric is defined as follows</p>
  <blockquote>
  <p>A metric space is a set <span class="math-container">$X$</span> together with a function <span class="math-container">$d$</span> (called a
  metric or &quot;distance function&quot;) which assigns a real number <span class="math-container">$d(x, y)$</span>
  to every  <span class="math-container">$x, y, z$</span> belongs <span class="math-container">$X$</span> satisfying the properties (or axioms):</p>
  <ol>
  <li><span class="math-container">$d(x, y) \ge 0$</span> and <span class="math-container">$d(x, y) = 0$</span> iff <span class="math-container">$x = y$</span>,</li>
  <li><span class="math-container">$d(x, y) = d(y, x)$</span>,</li>
  <li><span class="math-container">$d(x, y) + d(y, z) \ge d(x, z).$</span></li>
  </ol>
  </blockquote>
  <p>Do research papers generally use the word metric in the sense of the metric defined above? Or do we need to interpret the word metric less rigorously, just as a measure, like an accuracy?</p>
  <p>Note: Although I provided the abstract from a research paper containing the word metric, the question is not restricted to this particular context. This question can be applied to all AI-related research papers that used the word metric, especially in the context of performance or evaluation metrics.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-05T20:30:45.430Z
Comments.Text: I don't think you should interpret it rigorously. Most of the "metrics" used turn out to satisfy those constraints, but I think I've seen KL divergence-based "metrics" being called, well, metrics, while they do not satisfy property 2
Comments.UserDisplayName: Ant
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-06T14:01:43.570Z
Comments.Text: @Ant That looks like the beginning of an answer. In addition to KL divergence, maybe you can provide more examples of "metrics" that are or not really metrics by showing that they satisfy or not the properties of a mathematical metric.
Comments.UserDisplayName: nbro
Tags: <terminology><math><metric>
Title: Should I need to interpret the word "metric" in "performance metric" rigorously?
UserDisplayName: hanugm
ThreadText: # Should I need to interpret the word "metric" in "performance metric" rigorously?
  
  (Asked by: username_0 on 2022-04-05)
  
  Consider the following abstract from the research paper titled [A Note on the Inception Score](https://arxiv.org/pdf/1801.01973.pdf) for instance
  
  > Deep generative models are powerful tools that have produced
  > impressive results in recent years. These advances have been for the
  > most part empirically driven, making it essential that we use
  > high-quality **evaluation metrics**. In this paper, we provide new
  > insights into the Inception Score, a recently proposed and widely used
  > **evaluation metric** for generative models, and demonstrate that it fails
  > to provide useful guidance when comparing models. We discuss both
  > suboptimalities of the metric itself and issues with its application.
  > Finally, we call for researchers to be more systematic and careful
  > when evaluating and comparing generative models, as the advancement of
  > the field depends upon it.
  
  Here we can observe the usage of word metric several times. In mathematics, the word metric is used only in the context of metric spaces afaik. The [definition for metric space](http://www-groups.mcs.st-andrews.ac.uk/%7Ejohn/MT4522/Lectures/L5.html#:%7E:text=A%20metric%20space%20is%20a,(x%2C%20y)%20%3D%200) and metric is defined as follows
  
  > A metric space is a set $X$ together with a function $d$ (called a
  > metric or "distance function") which assigns a real number $d(x, y)$
  > to every $x, y, z$ belongs $X$ satisfying the properties (or axioms):
  >
  > 1. $d(x, y) \ge 0$ and $d(x, y) = 0$ iff $x = y$,
  > 2. $d(x, y) = d(y, x)$,
  > 3. $d(x, y) + d(y, z) \ge d(x, z).$
  
  Do research papers generally use the word metric in the sense of the metric defined above? Or do we need to interpret the word metric less rigorously, just as a measure, like an accuracy?
  
  Note: Although I provided the abstract from a research paper containing the word metric, the question is not restricted to this particular context. This question can be applied to all AI-related research papers that used the word metric, especially in the context of performance or evaluation metrics.
  
  **Tags:** terminology, math, metric
  
  > **Comments:**
  > **username_1**: I don't think you should interpret it rigorously. Most of the "metrics" used turn out to satisfy those constraints, but I think I've seen KL divergence-based "metrics" being called, well, metrics, while they do not satisfy property 2  
  > **username_2**: @username_1 That looks like the beginning of an answer. In addition to KL divergence, maybe you can provide more examples of "metrics" that are or not really metrics by showing that they satisfy or not the properties of a mathematical metric.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-05-16. Score: 1
  
  "Metric" should be understood as "a function of the trained model and of a dataset which returns a number".
  
  For example, in reinforcement learning, one can use as an evaluation metric the total cumulative discounted reward that a policy gets on a set of episodes for a given task, starting from the same initial state $S\_0$.
  
  That's a function $f(\tau\_1, \tau\_2, ... | \pi, S\_0)$, where $\tau\_1$ is the first trajectory obtained by running policy $\pi$ on the task, $\tau\_2$ the second one, etc. (we assume that $\pi$ is stochastic, otherwise it doesn't make sense to run multiple episodes). $f$ is an "evaluation metric" but is certainly not a distance function.
  
  
  ---
  
  