Answers.Body: <p>For the first question, RMSE and Euclidean distance have no difference, not that i know of. </p>
  
  <p>For the second question, you only need the common loss function for normal tasks. </p>
  
  <p>MSE is a common loss function used in <strong>linear</strong> regression tasks as well as loss function similar in nature like the RMSE. For classification tasks, Cross Entropy Loss is preferred. For <strong>logistic regression</strong>, use Binary Cross Entropy loss. See this for details:</p>
  
  <blockquote>
    <p>Cross-entropy loss, or log loss, measure the performance of a
    classification model whose output is a probability value between 0 and
    1. It is preferred for classification, while mean squared error (MSE) is one of the best choices for regression. This comes directly from
    the statement of your problems itself. In classification you work with
    a very particular set of possible output values thus MSE is badly
    defined. </p>
    
    <p>To better understand the phenomena it is good to follow and understand
    the relations between</p>
    
    <p>Cross-entropy</p>
    
    <p>Logistic regression (binary cross-entropy)</p>
    
    <p>Linear regression (MSE)</p>
    
    <p>You will notice that both can be seen as a maximum likelihood
    estimator (MLE), simply with different assumptions about the dependent
    variable.</p>
    
    <p>When you derive the cost function from the aspect of probability and
    distribution, you can observe that MSE happens when you assume the
    error follows Normal Distribution and cross-entropy when you assume
    binomial distribution. It means that implicitly when you use MSE, you
    are doing regression (estimation) and when you use CE, you are doing
    classification. </p>
  </blockquote>
  
  <p>Source: <a href="https://intellipaat.com/community/2015/why-is-the-cross-entropy-method-preferred-over-mean-squared-error-in-what-cases-does-this-doesnt-hold-up" rel="nofollow noreferrer">https://intellipaat.com/community/2015/why-is-the-cross-entropy-method-preferred-over-mean-squared-error-in-what-cases-does-this-doesnt-hold-up</a></p>
  
  <p>For a custom loss function, there is examples like the triplet loss and cases where you need to optimize two loss at once, then you need a custom loss function. For the case of triplet loss, it is used in one-shot learning tasks like face recognition. There, face images are fed through a CNN to get embeddings about description of the face. In the training phase there is three images per batch. One anchor, one positive (which is the same person as anchor
  ) and one negative (different person from anchor). The loss function, triplet loss, maximizes the distance between the anchor embedding and the negative embedding. In reverse, it also minimize the distance between the anchor embedding and the positive embedding. The loss function is like this:
  <a href="https://i.sstatic.net/WHdCj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WHdCj.png" alt="enter image description here"></a></p>
  
  <p>As you can see, there is two components here, and they are subtracted to maximize the second component and minimize the first. The alpha value is to make the loss positive as the gradient descent optimizes the loss to be as near to zero as possible. </p>
  
  <p>The case of style transfer also introduces a custom loss function. Here is the loss function:
  <a href="https://i.sstatic.net/lbjUE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lbjUE.png" alt="enter image description here"></a></p>
  
  <p>Style transfer basically alters the input image to be styled like the style image of the system.</p>
  
  <p>The loss function consist of style loss and content loss. The content loss decreases the distance of the output image and the style loss decreases the distance of the output image and the style in an artistic sense. The second loss makes the output image imitate the style of the the style image. </p>
  
  <p>In both scenario a custom loss function is used. In both cases, the custom loss function is used to optimize two losses together, either positively or negatively. A weight can also be introduced to weight the two losses as shown in the second case. However the second case introduce a custom loss function. That leads to the second use of custom loss function. They can be used for specific tasks and optimize a specific goal. The first use for custom loss function can be explored but the second requires careful engineering and research. </p>
  
  <p>So in short, custom loss function are used for either combining two loss or a specific task.</p>
  
  <p>Hope I can help you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T10:31:10.397Z
Answers.Text: thnx for your answer but with all respect I don't know how does that answer my questions. I mean I'm not a beginner with NNs,I already know the differences between mse/rmse and log loss and why is mse used for regression and cross entropy for classification. I also wrote in the question that I knew that. My question is about difference between rmse and euclidean direction and why would someone need to use custom loss if the standard losses works fine and are differentiable, why or when should I use a custom loss or how I get the intuition of using custom loss. that was my question
Answers.UserDisplayName: basilisk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T10:33:28.180Z
Answers.Text: Sorry didn't add the final part. Wait a second ediiting my question. Thank you for your reminder @basilisk
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T11:37:21.050Z
Answers.Text: I have changed the answer. Sorry for didn't adding it at first. Hope it can help you @basilisk
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T12:08:25.520Z
Answers.Text: thanks for the update. it surely helped to understand a use case for it, but although it does make sense to me, I can't understand a generalized idea of it. I understood from your answer that it is helpful to use that triple loss for image recognition and that can be used when I have 2 losses to optimize but  in the third paragraph I gave an explanation about my project which is a multi regression task with FFNNs or RNNs and my NN should predict two values that mean I ll have two losses I think, I don't think if the triple loss is useful for me, also my NN should minimize both errors
Answers.UserDisplayName: basilisk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T12:14:54.907Z
Answers.Text: so it ll not try to maximize one loss and reduce the other. I think probably a normal rmse loss can be applicable in this case, and that's  what actually the question is about. I think your answer is good and helpful but it point to a specific problem which is image recognition but if we talked in general, how or when and why does an ML Engineer decide to use a custom loss. let's say you are an ML engineer at work and you receive some project or task, how do you decide to tell your team for example hey people we should use a custom loss for this problem? I'm just curious about this
Answers.UserDisplayName: basilisk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T12:17:05.000Z
Answers.Text: you can also read the last paragraph of the question, I explained what is my project about and it's a research so no one have made this before and I read many research papers but there is nothing about it. My NN will give two outputs but it is not necessary that I use two losses because I want to minimize both those output losses so my case is not like the triple loss case you mentioned.
Answers.UserDisplayName: basilisk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T12:27:20.817Z
Answers.Text: So if a custom loss is used it is used for optimizing two different **goals** , not only optimize two things. The two goals maybe one positive one negative, or weighted loss. In your case of two output, you can just use a conventional loss function and optimize the output using the target as the two output have the same "goal" like the objective is to minimize the distance between the output and label, not one minimize one maximize. You don't need a custom loss function for this.
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T12:30:58.973Z
Answers.Text: thanks for the answer, I also thought so I just wanted to here another point of view from the community experienced people
Answers.UserDisplayName: basilisk
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T12:32:04.717Z
Answers.Text: Also since your problem is to predict the longitude and latitude, both outputs are the same kind, so no weighted loss is required.
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T12:36:56.303Z
Answers.Text: @basilisk I just wanted to add an example of a custom loss function. Ordinal Categorical Classification; meaning when order of the classes matter: given classes 1 to 5, identifying 1 as 2 is not as bad as identifying in as 5. There is no loss function for such a problem in Keras, so one has to use a custom function, or use regular classifier which do not use all the information available.
Answers.UserDisplayName: serali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-15T09:50:42.580Z
Answers.LastActivityDate: 2019-11-15T11:36:48.757Z
Answers.LastEditDate: 2019-11-15T11:36:48.757Z
Answers.UserDisplayName: Clement
Body: <p>I'm searching for a loss function that fits my project. Actually, I have two questions, but they are in the same direction. I take a look at the definition of the <a href="https://en.wikipedia.org/wiki/Root-mean-square_deviation" rel="nofollow noreferrer">root mean squared error (RMSE)</a> and the <a href="https://en.wikipedia.org/wiki/Euclidean_distance" rel="nofollow noreferrer">Euclidean distance</a> and they look the same to me. That's why I want to know the difference between the two. What would be the difference if I use RMSE as a loss function or the euclidean distance?</p>
  
  <p>The second question is how to search for a loss function. I mean I know it depends on the problem and common things are MSE for regression and cross-entropy for classification, but let's say I have a specific problem, how do I search for a loss function? </p>
  
  <p>I also saw that some people use a custom loss function and most of the deep learning frameworks allow us to define a custom loss function, but why would I want to use a custom one? How do I get <strong>the intuition that I need a custom loss function?</strong> </p>
  
  <p>To be more specific, I'm doing a project where I need to reduce the GPS error of a vehicle. I have some vehicle data and my neural network will try to predict the longitude and latitude, so it's a regression problem. That's why I thought that the Euclidean distance would make sense as a loss function, right? Now, somehow MSE also makes sense to me because it is getting the difference between prediction and ground truth. Does this make sense to you as a professional ML engineer or data scientist? And if there would be a <strong>custom loss function</strong> that you can use, what would you suggest and why?</p>
  
Tags: <neural-networks><deep-learning><comparison><optimization><objective-functions>
Title: What's the difference between RMSE and Euclidean distance, and when to use a custom loss?
UserDisplayName: basilisk
ThreadText: # What's the difference between RMSE and Euclidean distance, and when to use a custom loss?
  
  (Asked by: username_0 on 2019-11-15)
  
  I'm searching for a loss function that fits my project. Actually, I have two questions, but they are in the same direction. I take a look at the definition of the [root mean squared error (RMSE)](https://en.wikipedia.org/wiki/Root-mean-square_deviation) and the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) and they look the same to me. That's why I want to know the difference between the two. What would be the difference if I use RMSE as a loss function or the euclidean distance?
  
  The second question is how to search for a loss function. I mean I know it depends on the problem and common things are MSE for regression and cross-entropy for classification, but let's say I have a specific problem, how do I search for a loss function?
  
  I also saw that some people use a custom loss function and most of the deep learning frameworks allow us to define a custom loss function, but why would I want to use a custom one? How do I get **the intuition that I need a custom loss function?**
  
  To be more specific, I'm doing a project where I need to reduce the GPS error of a vehicle. I have some vehicle data and my neural network will try to predict the longitude and latitude, so it's a regression problem. That's why I thought that the Euclidean distance would make sense as a loss function, right? Now, somehow MSE also makes sense to me because it is getting the difference between prediction and ground truth. Does this make sense to you as a professional ML engineer or data scientist? And if there would be a **custom loss function** that you can use, what would you suggest and why?
  
  **Tags:** neural-networks, deep-learning, comparison, optimization, objective-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-15. Score: 3
  
  For the first question, RMSE and Euclidean distance have no difference, not that i know of.
  
  For the second question, you only need the common loss function for normal tasks.
  
  MSE is a common loss function used in **linear** regression tasks as well as loss function similar in nature like the RMSE. For classification tasks, Cross Entropy Loss is preferred. For **logistic regression**, use Binary Cross Entropy loss. See this for details:
  
  > Cross-entropy loss, or log loss, measure the performance of a
  > classification model whose output is a probability value between 0 and
  > 1. It is preferred for classification, while mean squared error (MSE) is one of the best choices for regression. This comes directly from
  > the statement of your problems itself. In classification you work with
  > a very particular set of possible output values thus MSE is badly
  > defined.
  >
  > To better understand the phenomena it is good to follow and understand
  > the relations between
  >
  > Cross-entropy
  >
  > Logistic regression (binary cross-entropy)
  >
  > Linear regression (MSE)
  >
  > You will notice that both can be seen as a maximum likelihood
  > estimator (MLE), simply with different assumptions about the dependent
  > variable.
  >
  > When you derive the cost function from the aspect of probability and
  > distribution, you can observe that MSE happens when you assume the
  > error follows Normal Distribution and cross-entropy when you assume
  > binomial distribution. It means that implicitly when you use MSE, you
  > are doing regression (estimation) and when you use CE, you are doing
  > classification.
  
  Source: <https://intellipaat.com/community/2015/why-is-the-cross-entropy-method-preferred-over-mean-squared-error-in-what-cases-does-this-doesnt-hold-up>
  
  For a custom loss function, there is examples like the triplet loss and cases where you need to optimize two loss at once, then you need a custom loss function. For the case of triplet loss, it is used in one-shot learning tasks like face recognition. There, face images are fed through a CNN to get embeddings about description of the face. In the training phase there is three images per batch. One anchor, one positive (which is the same person as anchor
  ) and one negative (different person from anchor). The loss function, triplet loss, maximizes the distance between the anchor embedding and the negative embedding. In reverse, it also minimize the distance between the anchor embedding and the positive embedding. The loss function is like this:
  [![enter image description here](https://i.sstatic.net/WHdCj.png)](https://i.sstatic.net/WHdCj.png)
  
  As you can see, there is two components here, and they are subtracted to maximize the second component and minimize the first. The alpha value is to make the loss positive as the gradient descent optimizes the loss to be as near to zero as possible.
  
  The case of style transfer also introduces a custom loss function. Here is the loss function:
  [![enter image description here](https://i.sstatic.net/lbjUE.png)](https://i.sstatic.net/lbjUE.png)
  
  Style transfer basically alters the input image to be styled like the style image of the system.
  
  The loss function consist of style loss and content loss. The content loss decreases the distance of the output image and the style loss decreases the distance of the output image and the style in an artistic sense. The second loss makes the output image imitate the style of the the style image.
  
  In both scenario a custom loss function is used. In both cases, the custom loss function is used to optimize two losses together, either positively or negatively. A weight can also be introduced to weight the two losses as shown in the second case. However the second case introduce a custom loss function. That leads to the second use of custom loss function. They can be used for specific tasks and optimize a specific goal. The first use for custom loss function can be explored but the second requires careful engineering and research.
  
  So in short, custom loss function are used for either combining two loss or a specific task.
  
  Hope I can help you.
  
  > **Comments:**
  > **username_0**: thnx for your answer but with all respect I don't know how does that answer my questions. I mean I'm not a beginner with NNs,I already know the differences between mse/rmse and log loss and why is mse used for regression and cross entropy for classification. I also wrote in the question that I knew that. My question is about difference between rmse and euclidean direction and why would someone need to use custom loss if the standard losses works fine and are differentiable, why or when should I use a custom loss or how I get the intuition of using custom loss. that was my question  
  > **username_1**: Sorry didn't add the final part. Wait a second ediiting my question. Thank you for your reminder @basilisk  
  > **username_1**: I have changed the answer. Sorry for didn't adding it at first. Hope it can help you @basilisk  
  > **username_0**: thanks for the update. it surely helped to understand a use case for it, but although it does make sense to me, I can't understand a generalized idea of it. I understood from your answer that it is helpful to use that triple loss for image recognition and that can be used when I have 2 losses to optimize but  in the third paragraph I gave an explanation about my project which is a multi regression task with FFNNs or RNNs and my NN should predict two values that mean I ll have two losses I think, I don't think if the triple loss is useful for me, also my NN should minimize both errors  
  > **username_0**: so it ll not try to maximize one loss and reduce the other. I think probably a normal rmse loss can be applicable in this case, and that's  what actually the question is about. I think your answer is good and helpful but it point to a specific problem which is image recognition but if we talked in general, how or when and why does an ML Engineer decide to use a custom loss. let's say you are an ML engineer at work and you receive some project or task, how do you decide to tell your team for example hey people we should use a custom loss for this problem? I'm just curious about this  
  > **username_0**: you can also read the last paragraph of the question, I explained what is my project about and it's a research so no one have made this before and I read many research papers but there is nothing about it. My NN will give two outputs but it is not necessary that I use two losses because I want to minimize both those output losses so my case is not like the triple loss case you mentioned.  
  > **username_1**: So if a custom loss is used it is used for optimizing two different **goals** , not only optimize two things. The two goals maybe one positive one negative, or weighted loss. In your case of two output, you can just use a conventional loss function and optimize the output using the target as the two output have the same "goal" like the objective is to minimize the distance between the output and label, not one minimize one maximize. You don't need a custom loss function for this.  
  > **username_0**: thanks for the answer, I also thought so I just wanted to here another point of view from the community experienced people  
  > **username_1**: Also since your problem is to predict the longitude and latitude, both outputs are the same kind, so no weighted loss is required.  
  > **username_2**: @basilisk I just wanted to add an example of a custom loss function. Ordinal Categorical Classification; meaning when order of the classes matter: given classes 1 to 5, identifying 1 as 2 is not as bad as identifying in as 5. There is no loss function for such a problem in Keras, so one has to use a custom function, or use regular classifier which do not use all the information available.  
  
  ---
  
  