Answers.Body: <p>A <a href="https://en.wikipedia.org/wiki/Loss_function" rel="nofollow noreferrer">Loss function</a> is just a function with a minimum.</p>
  <p>In machine learning though, we also require the loss function to be differentiable, otherwise no backpropagation and hence no weight updating. Moreover basically every deep learning library relies on autograd, so if the function is not differentiable your code will simply crash.</p>
  <p>Stronger but not compulsory condition could be <a href="https://en.wikipedia.org/wiki/Lipschitz_continuity" rel="nofollow noreferrer">Lipschitz continuity</a>, i.e. ensuring that the function decrease at a constant rate. Intuitively, a loss function should output high values for big differences between predictions/targets and small values for small changes, otherwise the update of the weights will risk to be too big (no convergence) or too small (easily stuck on local minima).</p>
  <p>Regarding your loss, the only issue I see is that you're replacing a proper metric, i.e. euclidean distance, with a function that is not a metric, i.e. cosine similarity (which does not respect the triangular inequality, hence it's not a metric). So I would be careful and test what kind of values you get with some dummy data to understand if it still behave as a proper loss.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-05T02:32:27.240Z
Answers.Text: Thanks heaps! That's exactely what I was looking for.
Answers.UserDisplayName: randomal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-04T11:53:02.663Z
Answers.LastActivityDate: 2022-04-04T11:53:02.663Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I am working on a modified version of the triplet loss function introduced with SBERT, where instead of the Euclidean distance we use the cosine similarity. The formula to minimize is <code>max( (|s_a*s_p| / |s_a|*|s_p|) - (|s_a*s_n| / |s_a|*|s_n|) + e, 0)</code> where <code>s_a</code> is the embedding of the anchor sentences (the context), <code>s_p</code> is the embedding of the positive sentence (correct continuation) and <code>s_n</code> is the embedding of the negative sentence (wrong continuation).</p>
  <p>I would like to check that the function I came up with makes sense from a theoretical point of view. Where should I look to check which features a loss function should satisfy?</p>
  <p>Motivation for the question: I'm getting my hands dirty with contrastive loss functions, and this is an easy variation I came up with.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-04T08:52:21.017Z
Comments.Text: You should share with us the loss function (the formula) that you came up with. Maybe you should also explain how it's different from the original loss and why we came up with this new one.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-04T09:11:47.213Z
Comments.Text: I modified the question as recommended.
Comments.UserDisplayName: randomal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-04T09:12:27.140Z
Comments.Text: Edit your post to include this info directly there. Note that you can use mathjax on this site.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-04T09:12:55.387Z
Comments.Text: I'm getting my hands dirty with contrastive loss functions, and this is an easy variation I came up with.
Comments.UserDisplayName: randomal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-04T09:22:05.950Z
Comments.Text: I modified the title to be more specific and to be the question that I think you're asking. Make sure that's the case. Again, I would highly recommend that you use mathjax to format the loss function.
Comments.UserDisplayName: nbro
Tags: <machine-learning><loss><triplet-loss-function>
Title: Does this modified version of the triplet loss function introduced with SBERT that uses the cosine similarity make sense?
UserDisplayName: randomal
ThreadText: # Does this modified version of the triplet loss function introduced with SBERT that uses the cosine similarity make sense?
  
  (Asked by: username_0 on 2022-04-04)
  
  I am working on a modified version of the triplet loss function introduced with SBERT, where instead of the Euclidean distance we use the cosine similarity. The formula to minimize is `max( (|s_a*s_p| / |s_a|*|s_p|) - (|s_a*s_n| / |s_a|*|s_n|) + e, 0)` where `s_a` is the embedding of the anchor sentences (the context), `s_p` is the embedding of the positive sentence (correct continuation) and `s_n` is the embedding of the negative sentence (wrong continuation).
  
  I would like to check that the function I came up with makes sense from a theoretical point of view. Where should I look to check which features a loss function should satisfy?
  
  Motivation for the question: I'm getting my hands dirty with contrastive loss functions, and this is an easy variation I came up with.
  
  **Tags:** machine-learning, loss, triplet-loss-function
  > **Comments:**
  > **username_1**: You should share with us the loss function (the formula) that you came up with. Maybe you should also explain how it's different from the original loss and why we came up with this new one.  
  > **username_0**: I modified the question as recommended.  
  > **username_1**: Edit your post to include this info directly there. Note that you can use mathjax on this site.  
  > **username_0**: I'm getting my hands dirty with contrastive loss functions, and this is an easy variation I came up with.  
  > **username_1**: I modified the title to be more specific and to be the question that I think you're asking. Make sure that's the case. Again, I would highly recommend that you use mathjax to format the loss function.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-04-04. Score: 1 | Accepted answer
  
  A [Loss function](https://en.wikipedia.org/wiki/Loss_function) is just a function with a minimum.
  
  In machine learning though, we also require the loss function to be differentiable, otherwise no backpropagation and hence no weight updating. Moreover basically every deep learning library relies on autograd, so if the function is not differentiable your code will simply crash.
  
  Stronger but not compulsory condition could be [Lipschitz continuity](https://en.wikipedia.org/wiki/Lipschitz_continuity), i.e. ensuring that the function decrease at a constant rate. Intuitively, a loss function should output high values for big differences between predictions/targets and small values for small changes, otherwise the update of the weights will risk to be too big (no convergence) or too small (easily stuck on local minima).
  
  Regarding your loss, the only issue I see is that you're replacing a proper metric, i.e. euclidean distance, with a function that is not a metric, i.e. cosine similarity (which does not respect the triangular inequality, hence it's not a metric). So I would be careful and test what kind of values you get with some dummy data to understand if it still behave as a proper loss.
  
  > **Comments:**
  > **username_0**: Thanks heaps! That's exactely what I was looking for.  
  
  ---
  
  