Answers.Body: <p><strong>Why we would like a function to satisfy some properties?</strong></p>
  <p>If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to converge. Other properties like symmetry and boundedness are not strictly required but they can help avoiding exploding loss scores as well as improving convergence time and convergence on <a href="https://arxiv.org/pdf/1901.09314.pdf" rel="nofollow noreferrer">corrupted labels</a>.</p>
  <p>If we're talking instead of a metric or similarity score then properties like having a unique maximum become compulsory. In the case of SSIM for example you want the function to return the maximum score only when comparing an image to itself. Symmetry is also a pretty obvious requirement, since comparing A to B should lead to the same score as comparing B to A.</p>
  <p><strong>Do AI-developers usually know properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?</strong></p>
  <p>From the paragraph above it should be clear that yes, these properties are relevant (not always essentials). A good developer should know which properties a function respect, many times it is as simple as taking a look at the function graph, otherwise referring to the original paper is always the safest option, also if you want to check if the authors bothered to mathematically prove that the properties always hold (which they almost always do if they want to get published).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-20T09:49:11.053Z
Answers.Text: Thanks a lot for the answer, it really clarifies my doubts. My only remaining question is the following: how do developers usually verify that the implementation of the function holds its properties (i.e., that it is implemented according to original paper's specifications)?
Answers.UserDisplayName: Theo Deep
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-20T11:32:05.730Z
Answers.Text: Unfortunately the only way is to go trough the code and check that it respect the math described in the paper. If a repo is well made tough, the code should include scripts to reproduce the paper results, which is enough of a proof that the implementation does what it's supposed to.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-20T08:25:31.240Z
Answers.LastActivityDate: 2022-09-20T08:25:31.240Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I will start with an example, in order to get to the general question.</p>
  <p>I was reading the following paper (<a href="https://www.cns.nyu.edu/pub/lcv/wang03-preprint.pdf" rel="nofollow noreferrer">https://www.cns.nyu.edu/pub/lcv/wang03-preprint.pdf</a>) about Structural Similarity Index (SSIM), which is a function used in computer vision. Basically, given two images, it returns (according to some criteria) &quot;how similar&quot; these images are.</p>
  <p>But what strikes me is that, in the paper, the following is stated: &quot;<em>we also would like the similarity measure to satisfy the following conditions</em>&quot;.</p>
  <p>I'll explain these properties now, but my question is the following:  <strong>why we would <em>like a function to satisfy some properties</em>?</strong> In other words, what I understand is that <strong>it is interesting to prove that these properties hold</strong>, am I right?</p>
  <p>Some years ago, I used SSIM not only as a metric to measure the performance of some algorithms, but also as a loss function itself. However, I definitively did not know about these properties, so it could be the case that I was optimizing (or measuring my results) with an implementation of the function that does not hold these properties, is that so?</p>
  <p>As for the properties, these are straightforward:</p>
  <ul>
  <li>Unique maximum:  <code>S(x, y) = 1 if and only if x = y </code></li>
  <li>Boundedness:  <code>S(x, y) ≤ 1 </code></li>
  <li>Symmetry:  <code>S(x, y) = S(y, x) </code></li>
  </ul>
  <p>So the problem of &quot;proving&quot; SSIM's properties is useful for me for to raise the next (more general) research question: <strong>do AI-developers usually now properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?</strong></p>
  <p>I would appreciate some insight about this.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-19T16:32:54.597Z
Comments.Text: If the function doesn’t measure anything of interest, what’s the point? Why not just output $7$ every time?
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-19T16:36:52.327Z
Comments.Text: I mean, the function has its own shape (with several calculi on covariances etc), but they also mention those three properties over the specified function. This is what I wonder: why specify some properties over an already-described function? How relevant are them?  Are the cases out there where these kind of properties not holding yield wrong optimizations/measurements?
Comments.UserDisplayName: Theo Deep
Tags: <machine-learning><objective-functions><loss><metric><ssim>
Title: Do we need to know or verify properties of loss functions / metrics' implementations?
UserDisplayName: Theo Deep
ThreadText: # Do we need to know or verify properties of loss functions / metrics' implementations?
  
  (Asked by: username_0 on 2022-09-19)
  
  I will start with an example, in order to get to the general question.
  
  I was reading the following paper (<https://www.cns.nyu.edu/pub/lcv/wang03-preprint.pdf>) about Structural Similarity Index (SSIM), which is a function used in computer vision. Basically, given two images, it returns (according to some criteria) "how similar" these images are.
  
  But what strikes me is that, in the paper, the following is stated: "*we also would like the similarity measure to satisfy the following conditions*".
  
  I'll explain these properties now, but my question is the following: **why we would *like a function to satisfy some properties*?** In other words, what I understand is that **it is interesting to prove that these properties hold**, am I right?
  
  Some years ago, I used SSIM not only as a metric to measure the performance of some algorithms, but also as a loss function itself. However, I definitively did not know about these properties, so it could be the case that I was optimizing (or measuring my results) with an implementation of the function that does not hold these properties, is that so?
  
  As for the properties, these are straightforward:
  
  * Unique maximum: `S(x, y) = 1 if and only if x = y`
  * Boundedness: `S(x, y) ≤ 1`
  * Symmetry: `S(x, y) = S(y, x)`
  
  So the problem of "proving" SSIM's properties is useful for me for to raise the next (more general) research question: **do AI-developers usually now properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?**
  
  I would appreciate some insight about this.
  
  **Tags:** machine-learning, objective-functions, loss, metric, ssim
  
  > **Comments:**
  > **username_1**: If the function doesn’t measure anything of interest, what’s the point? Why not just output $7$ every time?  
  > **username_0**: I mean, the function has its own shape (with several calculi on covariances etc), but they also mention those three properties over the specified function. This is what I wonder: why specify some properties over an already-described function? How relevant are them?  Are the cases out there where these kind of properties not holding yield wrong optimizations/measurements?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-09-20. Score: 2 | Accepted answer
  
  **Why we would like a function to satisfy some properties?**
  
  If we're talking about a loss function, you need to prove at the very least that the function has a minimum, otherwise you can't expect it to converge. Other properties like symmetry and boundedness are not strictly required but they can help avoiding exploding loss scores as well as improving convergence time and convergence on [corrupted labels](https://arxiv.org/pdf/1901.09314.pdf).
  
  If we're talking instead of a metric or similarity score then properties like having a unique maximum become compulsory. In the case of SSIM for example you want the function to return the maximum score only when comparing an image to itself. Symmetry is also a pretty obvious requirement, since comparing A to B should lead to the same score as comparing B to A.
  
  **Do AI-developers usually know properties of their loss functions/metrics? Are these properties relevant (e.g., are there functions for critical tasks)? Are they already being verified?**
  
  From the paragraph above it should be clear that yes, these properties are relevant (not always essentials). A good developer should know which properties a function respect, many times it is as simple as taking a look at the function graph, otherwise referring to the original paper is always the safest option, also if you want to check if the authors bothered to mathematically prove that the properties always hold (which they almost always do if they want to get published).
  
  > **Comments:**
  > **username_0**: Thanks a lot for the answer, it really clarifies my doubts. My only remaining question is the following: how do developers usually verify that the implementation of the function holds its properties (i.e., that it is implemented according to original paper's specifications)?  
  > **username_2**: Unfortunately the only way is to go trough the code and check that it respect the math described in the paper. If a repo is well made tough, the code should include scripts to reproduce the paper results, which is enough of a proof that the implementation does what it's supposed to.  
  
  ---
  
  